Fulton County Schools (FCS) is the largest school district in Georgia, with over 94,000 students and 11,500 full-time employees spread across 99 schools and 15 administrative buildings. The entire district uses a core school management system, or student information system (SIS), for all day-to-day student, faculty, and administrative operations. The system is used for enrolling students; registering them for classes; documenting grades, transcripts, results of student tests, and other assessment scores; building student schedules; tracking student attendance; and managing many other student-related data.
FCS had only three months before school started to migrate from its existing on-premises SIS to a completely new SaaS-based platform. It was to be a complete migration, as the two systems could not run concurrently. The SaaS platform had never been deployed on a school district as large as FCS. The district needed to be certain that when it cut over to the new SaaS platform, there would be no performance issues for the 2,000 concurrent users. Further, as the district continues to grow, FCS needed to be confident that the system could evolve with it.
“NeoLoad reduces test cycles from weeks to days.”
Under inflexible tight schedule to test brand-new core system end-to-end, NeoLoad cuts tests cycles from weeks to days
In May, FCS partnered with Total Performance Consulting (TPC) to design a comprehensive performance testing process, execute multiple rounds of testing, identify breaking points, and work with the SaaS platform provider to resolve scalability issues before school started in August. TPC selected NeoLoad as the performance testing solution that would enable FCS to launch the new SaaS school management system with a high degree of confidence — and implement continuous testing to ensure that as the district continues to grow, the system can grow alongside.
FCS had only three months before school started to migrate from its existing on-premises SIS to a completely new SaaS-based platform. It was to be a complete migration, as the two systems could not run concurrently. The SaaS platform had never been deployed on a school district as large as FCS. The district needed to be certain that when it cut over to the new SaaS platform, there would be no performance issues for the 2,000 concurrent users.
First, because the new SIS was a completely unknown platform for FCS — and responsible for so many different parts of school administration — TPC had to narrow the scope of the performance testing plan. It would not be possible to test everything, so TPC focused on the key activities with the highest impact on school operations.
Second, FCS needed to benchmark performance before and after the migration and find scalability issues. TPC designed a comprehensive testing process that mimicked usage in production of the existing SIS. That required a highly complex data management plan since the process of enrolling students, creating a family, selecting and registering for classes, removing courses, etc., involves large amounts of data that are shared and utilized multiple times. The performance testing plan included not only load tests for 2,000 concurrent users across many key use cases and scenarios but also spike tests and endurance tests. Because the SaaS platform had never been subjected to a deployment this big, the team anticipated many rounds of testing as TPC worked hand-in-hand with the SaaS platform vendor to test, monitor, identify, and resolve performance issues. The plan was on an aggressive three-month schedule (May to July) in order for the new system to be up and running with no performance issues when school started in August. The team was under the gun to perform a lot of complex end-to-end testing on an unknown platform in a very short period of time.
NeoLoad met all the challenges. First and foremost, NeoLoad enabled multiple rounds of very complex test cycles to run very quickly. This was crucial given the time crunch FCS was under and the amount of testing required for an unknown platform. TPC simply did not have the luxury of weeks long test cycles needed with other enterprise-grade performance testing tools. NeoLoad reduces test cycles from weeks to days. NeoLoad typically accelerates test cycles by 75% on average, due to its ease of use and automation features.
- Only three months to migrate from locally hosted school information system to SaaS-based platform
- Tight, inflexible deadline meant it needed to be operational by start of school
- Complete migration since the two systems could not run side by side
- New SaaS platform had never been used for school system this large
- Benchmark before-and-after performance on two completely different systems
- Performance test complex scenarios for 2,000 concurrent users
- Implement continuous performance testing to ensure system scalability as district grows
NeoLoad was able to test and benchmark performance for both the old on-premises SIS and the new SaaS-based SIS. FCS and TPC needed a tool that could create apples-to-apples comparison of performance between one on-prem system and another cloud-based system. TPC needed a solution that would help identify performance trends, pinpoint bottlenecks, and provide actionable intelligence for resolution. Through test execution, monitoring, and analysis, TPC was able to identify trends with the new SIS:
- Consistently high CPU on the database even at lower load levels
- Significant performance degradation and errors observed above 1,200 concurrent users
- Backend ETL (extract, transform, load) jobs having significant impact on database performance
- Search functionality significantly degrading as the number of users increases
TPC works with SIS vendor to resolve performance problems
TPC worked with the new SIS vendor to monitor the entire application stack during performance tests to understand the application and system bottlenecks. In addition to making some easy fixes, together they prioritized key problems in a three-step resolution path that included:
- Improving queries for search: Search functionality degraded significantly over 1,000 concurrent users. Reviewing and updating the stored procedures to make better use of indexes significantly improved performance.
- ETL jobs rescheduled: ETL jobs impacted performance by as much as 75 seconds for a single transaction. Rescheduling the jobs to off hours and separating the jobs that could not be rescheduled into smaller, more manageable jobs allowed for the performance impact to be negated.
- Database CPU: As the SaaS platform had never serviced a client the size of FCS, the database was not sized properly to handle the significant load FCS presented. Adding additional and higher performance CPU cores allowed the database to scale and remain highly performant at higher load levels.
FCS was able to find critical bottlenecks prior to migration to the new SIS platform. As a result, when the school year started, the initial rush of student registration did not cause any performance issues. With NeoLoad, the district now does continuous performance testing for new releases of the SIS platform. This allows FCS to maintain a high level of confidence that it will be able to continue to service its nearly 95,000 students and their families.
- Complete migration from core on-prem system to SaaS platform in only three months
- Pinpointed scalability issues and worked with platform vendor to resolve
- Problem-free new platform rollout, with zero performance issues, for first day of school
- Implemented continuous performance testing approach