Openboxes webUI slow loading on LAN with no WAN/Internet connection

[Richie_K]

Hello everyone,

I have encountered an issue with the Openboxes webUI, which is loading extremely slowly every time I disconnect my on-premise server from the Internet. Originally, my goal was to install Openboxes as an on-premise server that would be disconnected from the Internet and accessed locally via LAN (in this scheme: Server-Router-Client/Tablet). When I first installed Openboxes on my server, it was connected to the router, which had a connection to the Internet. When I accessed Openboxes using the IP:8080/openboxes, the login page loaded almost instantly. Also, browsing the rest of the webUI was with no issues. A few days ago, I unplugged the server from the Internet to test it out in the LAN setting with no WAN/Internet connection. When I accessed the login page the same way as before, the webpage loaded really slowly (almost 5 minutes) before it finally appeared. Sometimes it just timed out. And when I logged in and browsed the webUI, the loading took a few minutes, and sometimes the webpages appeared a bit messed up. So, my question is whether this is some kind of networking issue or if Openboxes (NGINX or Tomcat) uses some kind of dependencies that are downloaded every time from the Internet? Just to clarify, I’m using Linux server with v0.9.0-hotfix1 (Grails 3.3.16) version of Openboxes with NGINX web server configured to run on localhost:8080 and I was accessing it via a wireless router using the server IP (I´m checking my server IP everytime with the network scanner, haven´t set it to static yet). I also tried using multiple routers with the same results. I also tried disabling the firewall in my router with no success.

[David_Douma]

I think the issue is that some bootstrap content like Jquery is loaded off side, like:

https://cdn.jsdelivr.net/npm/bootstrap@4.3.1/dist/js/bootstrap.min.js
https://cdnjs.cloudflare.com/ajax/libs/datatables/1.9.4/jquery.dataTables.js
https://use.fontawesome.com/releases/v5.14.0/js/all.js

[jmiranda]

This is a great question and something we need to discuss internally to see if there’s a decent solution. I would like OpenBoxes to be usable on a LAN but we have not made any effort to ensure that it can.

At the moment, you probably won’t be able to run OpenBoxes via the browser on computers that have never had access to the Internet since there are resources (javascript, CSS, fonts, etc) that are being pulled from CDNs when pages are loaded for the first time (as @David_Douma suggested).

You are currently in a position where OpenBoxes on your computer mostly works for you because you were previously connected to the Internet. In other words, those resources were downloaded and cached by your browser. As soon as you clear your cache you’re going to start seeing problems.

We can definitely pull all resources down into the WAR file and forgo the need for CDNs. We just haven’t prioritized that.

The particular issue regarding slowness is related, but probably a slightly different problem which you should be able to confirm easily by using the browser’s Inspect feature.

The first time you load a page you’ll get all resources downloaded from their respective sources (local server, CDNs, etc)

On subsequent page loads, you’ll notice a bunch of resources under the Network tab with “(memory cache)” or “(disk cache)” in the Size column. These are the ones that’ll be fine for a period of time since they’ll continue to be loaded from the browser cache. But once you clear the cache, then you’ll need the internet again for anything that has been downloaded from a CDN.

In addition, you’ll also notice a bunch of resources that continue to be downloaded each time (nr-spa-1.252.1.min.js) along with requests that are made to services we use for application monitoring and .

For example, the POST request to NRJS-643a83ee6a48748926e is sending metric data to NewRelic to help us identify performance issues.

POST https://bam.nr-data.net/events/1/NRJS-643a83ee6a48748926e

a    = 600299440
v    = 1.252.1
to   = MwAGY0tXWkQFUUdaWwpKS1hJU1pVC0pWQBsFEBBfFlpbUA1c
rst  = 828
ck   = 0
s    = e94758a5c21d5205
ref  = https://obdev3.pih-emr.org/openboxes/auth/login
ptid = 5ff39916-0001-bfee-cd38-018e33ce095e
hr   = 0

As far as I recall, most of these features (HotJar) are disabled by default, so they should not be causing you any issues. That is, except for the New Relic requests. But, I was told by a developer that these requests would not impact performance as they should be happening asynchronously.

I’ll try to add this topic to our tech huddle for tomorrow and see if we can come up with a strategy and perhaps a short-term fix. If you can pinpoint the slow-loading requests then we might be able to get a fix in before the next release.

Here’s how to take a screenshot showing all requests being loaded for a particular page.

1. Ensure Internet is not enabled on your local machine
2. Open your browser
3. Navigate to OpenBoxes login page
4. Right click anywhere in the page
5. Select the Inspect option
6. Click on the Network tab
7. Refresh the login page (in order to download the resources for that page)
8. Take a screenshot

The most important data to capture (besides the Name) will be Time, Size, and Waterfall, so please make sure those columns and their values are visible.

You can also download the full log by clicking the download arrow (Export HAR…) in the top bar of the Network tab.

[Richie_K]

Thank you, gentlemen, for your responses. I will try to simulate the same conditions under which the loading slowdown occurred (i.e. without internet and with a clear cache) and will add screenshots from the browser network monitor here along with the HAR log.

[Richie_K]

Here are the results after I disconnected from Internet, went to the Login page in Mozilla, cleared cache and then refreshed the page with network monitor running. It was loading as slow as the first time when I encountered this issue. I hope it contains all the necessary data.

This is a mashup from all those three screenshots for better overview

[jmiranda]

Yeah, the logs are showing part of what I expected … that you’re encountering request timeouts for each of those CDN resources. But I don’t see the New Relic requests so I assume that feature is disabled by default.

/me goes off to look at our production server

Yup, looks like New Relic is only enabled if you set all or some of these command line arguments.

jmiranda@obnav:~$ ps -ef | grep tomcat | grep newrelic
...
-javaagent:/var/lib/tomcat85/newrelic/newrelic.jar 
-Dnewrelic.config.application_logging.forwarding.context_data.enabled=true 
-XX:StartFlightRecording 
-Dnewrelic.config.jfr.enabled=true 
...
org.apache.catalina.startup.Bootstrap start

So, in short, the New Relic agent is not the problem. The CDN resources are the problem. And without refactoring our code to include all resources within the WAR file, there’s probably no chance you’ll be able to get your implementation working without Internet.

I brought this issue up during our Tech Huddle this morning and was told that it’s not a priority for the organization funding development of OpenBoxes and there’s a lot of tech debt ahead of it in the backlog, so this probably won’t be prioritized for any of the upcoming releases (v0.9.1, v0.9.2, etc).

So you have one of three or four options

1. Wait until our development team can prioritize it (which might be months)

2. Hire your own developer to contribute a patch for this improvement

3. Fund the development through us (I’ll either work on it myself or find a developer to work on it)

4. Abandon OpenBoxes altogether

If you want to look into options (2) or (3) I can investigate the issue further and try to estimate the hours I think it might take. That’ll give us a ballpark cost estimate.

As I see it, there are a few issues that need to be addressed (potentially more once you get to requirement 3).

1. Should require all resources (js, CSS, fonts) to be included in the WAR file and not served from remote CDNs

2. Should allow disabling of all dependencies / features that require Internet (NewRelic APM, barcode label generation). We might actually want to add a configuration property to make this easier.

   openboxes.security.lockdown.enabled = true
   

3. Should investigate what else might be causing slowness once resources are pulled in locally

4. [optional] Should allow failover to a configured endpoint when requests timeout

   • Load resources locally instead of CDNs
   • If this works, we could either add a small improvement to the application to load a custom JavaScript file from the filesystem on your serve. That would allow you to experiment with a failover feature like the one in that article.
   • Unfortunately, that would be the easy part of the solution. The difficult part would be building your own local CDN server (see below).

<aside>

Aside: I just thought of one (somewhat painful) path you can take without code changes. You could create your own CDN-like server locally that can serve all of these resources and provide mock responses to requests for external APIs that we might be calling.

In a nutshell, you’d need to do something like this

1. Enable Internet

2. Download each of the resources required by the application (or fetch them from the browser cache)

3. Create a local CDN server using NGinx or Apache (or an open-source CDN software)

4. Configure local DNS to resolve all external hosts to the local CDN server

     Oh my, I'm sweating just thinking about how horrible this is ...
   

5. Configure virtual hostnames for all of the external CDNs

6. Load content into directories

7. Ensure paths in the request URL are maintained (should be configurable within Nginx or Apache or you can create the same directory structure under each CDN’s document root)

NOTE: There might be a more straightforward / automated way of doing this, but that is outside the scope of my expertise.

</aside>