BACKGROUND
California’s Coachella Valley Water District (CVWD) was formed in 1918 to protect and conserve local water sources. Since then, the district has grown into a multifaceted agency that delivers irrigation and domestic (drinking) water, collects and recycles wastewater, provides regional storm water protection, replenishes the groundwater basin, and promotes water conservation. Today, CVWD’s service area covers approximately 1,000 square miles from the San Gorgonio Pass to the Salton Sea, mostly within the Coachella Valley in Riverside County, California. District boundaries also extend into small portions of Imperial and San Diego counties. CVWD’s Information Technology (IT) department was recently awarded with The 2020 CIO 100 Award which honors 100 innovative organizations that exemplify high levels of operational and strategic excellence in IT. The award celebrates CVWD key innovations created while implementing a supervisory control and data acquisition (SCADA) master program. A key innovation included establishing a new control room and data center.
CHALLENGE
The CVWD’s requirements for its main control room include overseeing data acquisition, alarms, and control for 500 radio-controlled field sites – wells, sanitation lift stations, and other facilities – across the district’s 1,000-square-mile area, as well as a 230-mile canal and three large wastewater plants. To replace a control room that was installed more than two decades ago and run on old building management systems, the CVWD worked with Integrated Media Systems to construct a new main control room and a significant update of both its software and hardware systems. A critical project goal for the CVWD was to implement flexible, secure, efficient, ergonomic, and intuitive solutions for signal extension and switching, as well as video and data display, within the new control room.
SOLUTION
To ensure the long-term utility of its new main control room, the CVWD opted to deploy 4K-capable KVMover-IP and video wall processors from Black Box. The CVWD started the project by building a temporary control room adjacent its existing control room in Coachella. Built entirely on Black Box KVM and video wall systems, the temporary control room connected to existing control room infrastructure to keep operations running smoothly. The temporary facility served as a proof of concept for the main control room built in Palm Desert, and it supported testing of not only software, but also of desk lifts, monitor resolution and extension distance, and the ergonomics of the overall installation. “Our temporary control room got used so much — with the Black Box touchscreen, all the video sources, and the KVMs being able to control multiple machines — that the room wound up being far more than just a prototype,” says John Hopper, SCADA systems analyst at CVWD. “It’s now become a fully functional little control room.” The CVWD’s brand-new 24/7 main control room is now located in the district’s Critical Services Building, an entirely new building in Palm Desert that also houses a data center, water quality labs, office spaces, meeting rooms, and an emergency operations center (EOC)/training room. The new SCADA control room is the central point of monitoring and control for CVWD systems and resources. Integrated Media Systems was responsible for the control room facility design and build, as well as for full systems pre-installation assembly, control system programming, testing, and on-site installation and commissioning, all of which was completed early in 2020.
The state-of-the-art control room is equipped with three operator stations and two additional workstations that are supported by one 2x8 and two 2x2 video wall systems. A Black Box Radian Flex processor with an expansion chassis provides multi-image processing for all these displays, and the Emerald KVM system from Black Box supports operator PC control and display extension. Via AV control systems at their stations, operators can access Radian’s preset situational multi-image layouts for all video wall areas, and even share them with the other rooms in the facility. Using the networked Emerald KVM system to access the Radian processor, supervisory personnel can modify and customize layouts and sources. The Black Box systems allow operators to connect to an enterprise computer for regular district business and to the SCADA systems that enable gathering, monitoring, and processing of real-time data from devices deployed across the district. The KVM system also connects to remote computers to give operators visibility into and control over security cameras and door access. In the control room and in several of the building’s offices, the Black Box Freedom II KVM system with Glide & Switch mouse switching gives operators the flexibility to use just one keyboard and mouse while working across multiple computers. In case of an emergency event, the SCADA control room, emergency operations center/ training room, and three conference rooms will become integrated collaborative spaces with shared content throughout the facility. “The CVWD application is a complete communication solution.” says IMS President and CEO Brad Caldwell. “It provides stand-alone room functionality to support day-to-day operations and, with onetouch selection of emergency mode from the AV control system, fully integrated collaborative capability across the control room, EOC/training room, and conference rooms.” Eventually the CVWD will also retrofit the original control room (in Coachella) with Black Box equipment in a duplicate of the temporary control room. The original control room in Coachella has some radio equipment that is still in use and can’t be moved. So, Hopper and his team are using Black Box Emerald transmitters and receivers to acquire monitor, mouse, and keyboard signals from those old systems, convert them from copper to fiber, and deliver them via microwave link from Coachella to Palm Desert to small screens in the new control room until all of the 500 field sites have been updated. A high-speed network connects the Coachella and Palm Desert facilities, and the microwave link provides redundancy, making it possible to run the entire operation from either facility.
RESULTS
“The CVWD’s transition to this new state-of-the-art control room was a complex undertaking, and we’re proud of the results,” says Caldwell. “We worked closely with the CVWD and partners, including Black Box, to deliver critical functionality and to ensure intuitive operation. The installed solution reflects the creativity, innovation, and ‘can do’ attitude of our design and technical teams.
The Black Box solutions deployed across the CVWD’s temporary and brand-new control rooms give the district more flexibility in signal management and allow for much better optimization of workstations and workspaces. While the IP-based extension and switching of signals offers immediate benefits, including simpler integration with third-party systems for seamless communications and control, 4K readiness will allow the district to shift to higherresolution images and video in the future. Radian Flex video wall software supports the display of high-quality content from any source in countless configurations, including single images across multiple screens or multiple images on single screens. Using their touch screens, operators simply push a preset to configure displays throughout the entire room. Typically, the configuration is set as a large, level-1 awareness display so that with just a glance up at the large-scale display, operators can get a complete picture of the district’s status. “You know how hard it can be to get just five people to agree when ordering pizza? We have 10 operators, with two or three on duty at any time,” says Hopper. “Thanks to the flexibility of Radian Flex, it wasn’t a big deal. I was able sit in the room at a remote client and configure the templates and the presets, let them test it out on the fly, save the settings that worked for them, and we were done.” Accounting for roles, personal preference, and whether operators are left- or right-handed, Hopper created multiple presets that can be called up instantly by the operators on shift. The software is easy to use, so operators have learned how to create and adjust their own templates as needed.
The Emerald KVM-over-IP system delivers pixel-perfect video and facilitates reliable extension and switching of video, USB 2.0, and bidirectional analog audio signals to provide desktop access to computers and servers stored remotely. “For as complex as this installation is, it’s amazing how simple configuration of the receivers and transmitters was,” notes Hopper. “I was able to learn the Emerald KVM system quickly, and the support through both Integrated Media Systems and Black Box has been spectacular. I’ve never experienced that level of help from tech support.” In addition to reducing clutter, the Emerald system offers reliability that is essential to mission critical consoles within the control room. When the power accidentally went off on all racks during installation, the Emerald systems came back up without issue. “The nice thing about the Emerald gear is that if you do have switch problems or any other issue, you can simply patch the receiver to the transmitter, and your people are up and running,” says Hopper. “So, no matter how bad the situation gets, I can have consoles working in under 10 minutes.” Backed up by the temporary control room in Coachella, the new CVWD main control room in Palm Desert handles the district’s domestic water supply, as well as several large water treatment plants, and it distributes reclaimed water to all the golf courses. Overseeing 120 miles of canals with gate structures, the facility also controls water for farming and irrigation.