Simulcast, Chapel Style

I’ve wanted to do a post for a while about how we do our Simulcast services at The Chapel, so here’s how we do simulcast, Chapel style.

What is our definition of “Simulcast” you may ask? Well, it is a “Live” remote viewing of our center and side screens. I put “Live” in quotes because we time-slip the service on some DVR’s first to give us some flexibility on playback. Usually it runs 2 to 10 minutes behind live.

So, here is the list of equipment involved in capturing, encoding, sending, receiving, decoding, and playing back our Simulcast feed.

• Capture – Cameras are Sony PMW-EX3
• Encode/Decode - HaiVision Hai1060 chassis with two MAKO-HD cards
• Transport - All Cisco brand switches and routers over a 25 Mb Opt-E-Man circuit from AT&T
• Record – 360Systems for the Center HD center feed and Sony DSR-1000 for the SD side
  

Before we get into more technical stuff, let me explain why we do Simulcast. Shortly after moving into our new building in 2004, we started having talks about “Phase 2” of expansion to fit all the new people who were coming to The Chapel. This is a great problem to have but we hadn’t expected to have it so soon and didn’t have the money to expand quite yet. We also didn’t like the idea of more and more people driving farther and farther to go to church each week.

About that time our Sr. Pastors started having some talks with some struggling churches in the area. Two had approached us and wanted to join what God was doing through The Chapel. At that time, we also receive a large donation specifically for our multi-site campaign. A church in Barrington had built a new campus and had their old building up for sale. So, the march was on and we went from one church in one location, to one church in 4 locations in one year!
The first year we did a tape/dvd delay for one of our campuses and did “sneaker net” transport. This worked alright but the quality just wasn’t there. Also, our Sr. Pastors were getting worn out running between the 3 other campuses each weekend. Three of our campuses were also running a week delay in service but this caused other problems for our tech teams and our church members as well.

So, that is what lead up to us going to a live video simulcast on the weekends. Our pastors still preach 4 times a weekend (Twice on Saturday and twice on Sunday) but with our 4 campuses, that comes out to 10 services! Our pastor’s alternate teaching from our two large campuses which are Grayslake and Libertyville. One week, it will be live at Grayslake at the 9:00 am and simulcast everywhere else, and then it will be live at Libertyville for the 11:00 am and simulcast everywhere else. Then we switch the next weekend. Still crazy but much better than the alternatives!

So, onto the tech details that you all want to know. For cameras we use two Sony PMW-EX3 for the side iMag shots and one Sony PMW-EX3 with a different lens for the fixed center shot. All three of these cameras are HD, but we only project HD 1080i on the center screen at Libertyville and Grayslake. The screens are just so large we had to go HD to get the picture quality. The sides are Sanyo projectors that shoot 720p at Grayslake and Libertyville, and center and sides at Barrington and Mundelein.

Once the cameras capture the live service, it gets piped though what seems like an endless sea of cables and devices and then gets encoded and sent out to our other campuses. The gear we use for encoding and decoding our video is from HaiVision. At each campus we have a HaiVision 1060 chassis with two of their MAKO-HD cards. This system lets us do two simultaneous HD video streams which come out to about 15 Mbps total. Currently we send 1080i and it gets scaled at the decode sites to match the projectors.

This signal then gets sent out onto the network to our other sites on multicast addresses to reduce network traffic. We have a 25 Mb Metro-Ethernet link from AT&T between each site which has worked well for us so far. We did have to make sure our internal network was rock solid and properly configured, and work out some issues with AT&T before we were able to get a video to stay stable for up to an hour.

At the receive sites, we have two DVR’s that capture the service. We use a unit from 360Systems to capture the center HD signal and a Sony DSR-1000 that records the scaled down SD side feed. There is a DNF that is used to manually synchronize the two feeds for playback. Synchronizing the feeds is one of the hardest parts of the whole process because if you are off by even a few frames, people can tell.

For our backup, we capture the sides of the Saturday service at Grayslake or Libertyville on a Mac Pro with an HD capture card. Once the service is over on Saturday night, the video file of the side screens gets sent over the network to the other sites for a backup just in case.

So that’s it. That is how we roll. I’ll have a post coming shortly about some Cisco IOS magic we had to work to make sure our network was up to the challenge of multicast video, voice, and data. If you have any questions or comments I would love to hear them. Our simulcast solution is constantly being improved and I’ll post any new developments when they happen.

Internet Failover Using EIGRP

Internet Failover Using EIGRP

If you have a mostly Cisco network like we do, you know that there is nothing you can’t do, for a price. Cisco has some great Fail-Over techniques using their ASA 5500 series Firewalls but you need to get the extra HA licensing. Also, what do you do if your other firewalls aren’t Cisco? In this case, EIGRP can get you the same basic features for free. Now before I get lots of comments on this, using dynamic weighted routes with EIGRP does not provide Stateful Failover meaning that you will drop your connection for a short period while the network converges and you won’t be able to setup BGP this way (I think).



The Setup

For this post, I’m going to use a simple scenario of having two sites with two internet connections. One is much faster than the other and it makes sense to backhaul the internet from Site B to Site A when the connection at Site A is active. For this post, let’s assume that you want to just protect against the link between sites going down and not things like the Internet modem locking up (In future posts I’ll address this).

RTRA EIGRP Config

RTRA Fa 0/0 ip address is 10.100.1.2 255.255.255.0
RTRA Fa 0/1 ip address is 10.1.1.1 255.255.255.0
FWA Fa 0/1 ip address is 10.1.1.254 255.255.255.0

RTRA# conf t
RTRA## ip route 0.0.0.0 0.0.0.0 10.1.1.254
RTRA# router eigrp 100
RTRA# network 10.1.1.0 0.0.0.255
RTRA# network 10.100.1.0 0.0.0.255
RTRA# redistribute static metric 20000 1 255 255 1500

RTRB EIGRP Config

Fa 0/0 ip address is 10.100.1.3 255.255.255.0
Fa 0/1 ip address is 192.168.1.1 255.255.255.0

RTRB# conf t
RTRB# ip route 0.0.0.0 0.0.0.0 192.168.1.254 255
RTRB# router eigrp 100
RTRB# network 192.168.1.0 0.0.0.255
RTRB# network 10.100.1.0 0.0.0.255

Summary

Both routers are participating in EIGRP group 100. They will advertise their routes to each other and calculate the metric based on bandwidth, congestion, etc. The key here is setting the “redistribute static metric 20000 1 255 255 1500” command on RTRA. This tells EIGRP to redistribute the static routes you have setup in the router with a bandwidth of 20,000 kbps, delay of 1, reliability of 255, loading of 255, and MTU of 1500. RTRB has a static route for 0.0.0.0 but with a metric of 255. The dynamic route from RTRA will have a metric of 170 (External EIGRP) and will replace the static route in the routing table. This works fine if the only static route you have is for your default route so this isn’t very flexible. You could also attach a summary default route to Fa 0/0 and accomplish the same thing.

If the link between RTRA and RTRB is active, traffic that doesn’t have a route in RTRB (Internet) will be sent to RTRA. In the event that the link between RTRA and RTRB goes down, RTRB will send traffic to its local internet connection because the static route is now the only default route.

Now the biggest downside to this type of “failover” routing is that it isn’t fully dynamic and doesn’t take into account the fact that Cable or DSL modems can lock up and will still look like they are up to the firewall or router and therefore not remove the route. This approach also doesn’t work well if you have even more locations with their own Gateways to the Internet. I will address this in the next post.

Frozen!

We are getting closer to having our fiber network installed. For the last week we have seen many AT&T trucks around our Grayslake campus pulling fiber optics up and down the highway. The fiber finally arrived yesterday but there was one big problem that prevented it from being connected.

ICE!

We had rushed to get the conduit on our property run this fall so that we wouldn't hold up the project. We found out yesterday that that conduit had filled with water over the last two months! The record low temperatures we have had over the past month had caused a small portion of our underground conduit to freeze! Now we had to figure out how to clear it out. Hopefully without digging it up.

This is what we tried:

1. (Prayer)
2. Fish tape. Nope
3. Bigger fish tape. Nope.
4. Power rodder (for sewer pipes). Nope.
5. Garden hose run down the pipe to the blockage and hot water turned on. Success!

After leaving the hose on for about 30 minutes we had melted though the 2' section of ice and the pipe was clear. We pumped and vacuumed the water out and sucked a new rope though. Later today, AT&T came back and pulled the fiber.

Hopefully soon our fiber will be done and we can communicate at the speed of light!

Comcast for Mundelein

Can you believe that only one of our four sites can get Comcast? Turns out it is our Mundelein campus so we jumped on the option. Initially Comcast told us that they would have to trench from the back of our property to the building across our new parking lot. I was hoping that would not be the case.

I met the installer out there on Friday to discuss what we were going to do to get the service to the building. He said that it was already there. Confused I had him show me and sure enough there was a green pill shaped junction box right next to our power transformer! I had assumed that it was a phone junction!

So, after a quick install we now have 16 Mbps down and 2Mbps up Internet for about $100 a month! After some speed tests I found that it was about 14 Mbps down and 8 Mbps up! This is going to help our Internet needs tremendously once our Opt-E-Man fiber network is in place. Going to test with the 5 port SMC modem/router they provided but then need to look for a better firewall. Looking at SonicWall this time.

In the distance, its, Opt-E-Man!

Soon we will have fiber run to all four of our sites! Libertyville and Mundelein are all set and Barrington should finish today. That just leaves Grayslake and we will have our end of the work done. Then it is just up to AT&T to get rolling on the fiber installs.

This will really open some doors in the future having fiber installed. We will be able to expand our bandwidth by just making a call! Of course we are at AT&T's mercy with pricing and turn around. We can only hope they will be a little quicker once the fiber is installed.

High Speed

Things have been moving along with our Opt-E-Man installation. The directional boring has been done at Libertyville and Barrington and the trenching at Mundelein is scheduled for tomorrow. After we get our conduit done we will just be waiting on AT&T.

We will also be getting Comcast cable Internet at our Mundelein campus because it is the only one that can get it. More details on that when we get it installed.

Also, today our Communications team now all have Gigabyte access to our network in their new office space. We are using an 8 port Dell gig switch and it seams to be working well.