Developing a Web Scraper

Admin Dashboard

Once the key components were constructed, it became clear that an administration dashboard would be helpful for evaluating the performance. The dashboard allowed the developer to invoke the scraping, monitor its progress, examine the output, and compare it to what was expected. The Dashboard thus became the primary interface for identifying flaws in the logic and/or layouts and styles. Using a test environment further made it not only plausible, but desirable to enable all error and warning reporting making it easier to diagnose the severity of any issue. It should be noted that in a live production environment, it is common to suppress such errors to allow the program to gracefully exit or redirect without alarming the end user. The Dashboard further became the ideal interface for running analytical reports. A screenshot of the dashboard is shown in Figure 2.

Figure 2: Admin Dashboard