FAC system status

March 25, 2024

Search is now fully restored and available for users. We split search functionality into two pages:

• Basic search: supports real-time search. We recommend this page for auditors and auditees who would like to confirm their submissions are complete.
• Advanced search: supports search features typically required by federal users. Updated nightly.

If you have questions about search, please contact the FAC helpdesk.

March 22, 2024

Our engineering team continues to test the new infrastructure supporting search.

We are continuing to upload static lists of submitted reports daily for audit resoluiton officials.

If you have questions about search or the audit resolution lists, please contact the FAC helpdesk.

March 21, 2024

Testing of the new search infrastructure is still underway but our engineering team is making progress. We expect to turn search back on by early next week.

Again, we want to caution users that you will experience slow load times. We expect searches using multiple filters to take several minutes.

We now have static lists of submitted reports available. These exports list all audits with awards findings submitted by date.

If you have questions about search or the audit resolution lists, please contact the FAC helpdesk.

March 20, 2024

Our engineering team has made progress on improving search stability and performance. We'll be turning back on all search filters for testing soon. During this testing period, we want to caution users that you will experience slow load times. We expect searches using multiple filters to take several minutes.

If you experience a system error message, please contact the FAC helpdesk with the details of your search.

We're also continuing to work on a static list of submitted reports for use by audit resolution officials.

March 19, 2024

We're aware that search performance, even reduced to UEI-only, continues to be slow. Our engineers are investigating the root cause of the issue and continue to work on a solution.

We are also exploring options for using the API to generate a static list of submitted reports for use by audit resolution officials. We'll post more on this page if this becomes available.

For specific questions or issues, please contact the FAC via our helpdesk.

March 18, 2024

Audit search is currently limited to UEI-only. You can search your UEI to confirm your audit submission is complete. The PDF report is still be available for download but summary report workbooks aren't.

Issue background

In mid-February, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system exponentially, hindering search performance.

Over the last month, our team has released multiple patches to stabilize search with limited success. Because of this, we've reduced search to UEI-only while we focus on building a more sustainable search function.

We're working hard to return full search functionality as soon as we can. We know this is a significant disruption for many users. Thank you for your patience.

A timeline of the issues:

• February 7: We completed the data migration. This increased the number of audits from 10K to more than 250K, and the number of awards tracked from 100K to nearly 4.5M.

• February 8: We increased resources in our production environment to address the increased load.

• February 9: The team removed the My ALN filter in an attempt to stop system crashes caused by this feature.

• February 14: We worked with more than a dozen NSACs to prioritize search needs.

• February 16: The team increased search capacity in the production environment.

• February 28: We added prioritized search needs, re-introducing performance challenges.

• March 11: Heavy use of the system prevented system updates for nearly a week. The team completed an after-hours update to the production environment, partially addressing new search performance issues.

• March 14: Performance issues and system crashes persisted in search. The team released new search tables and queries. These required significant engineering and testing work.

• March 18: The team decided to take down search to focus on long-term solutions vs. short-term patches.