Problems Restoring UniFi Controller
I recently had to rebuild the Raspberry Pi that runs my UniFi controller, and had a much more difficult time than I expected. I would love to say that this is a complete log of the steps that I took, but alas, it probably is not. In any case, the problem appears to be that the default “new” user interface will not restore a backup, so all of the steps that I took prior to figuring that out probably were not necessary.
Once I changed to the “classic” user interface, my backups restored without incident.
Java 8, Not Java 9
The default Java installation on Raspian was Java 9, with a
/usr/lib/jvm/java-9-openjdk-armhf. UniFi doesn't work on Java 9. Make sure to install and Java 8 and set
/usr/lib/jvm/java-8-openjdk-armhf. Since this is the only thing that uses Java on this machine, I just created a file
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-armhf export PATH=$JAVA_HOME/bin:$PATH
UniFi 6.4.54 Would Not Restore Backup During Install
I placed the old backups in both
/usr/lib/unifi/data/backup/autobackup where all of the documentation said that they should go, and in
/var/lib/unifi/backup/autobackup where I actually found them on the old server. The newly-installed UniFi controller would not find them during initial login. It would not do anything when I tried the upload option.
Next, I tried created a new controller login and then tried to restore from a backup, but it still would not find them. Only when I changed permissions to 644 would it find the old backups, which is a big security problem. It would churn on a backup for a while, but would not restore it.
I then reverted to the old user interface in hopes of getting more diagnostic info, and my backup then restored on the first try. I won't be trying the new interface for a while.
- Written by Bruce Moore
- Hits: 690
Virtual School Crashed Our Spectrum Internet Connection--What Now?
We've had a 100 Mbps Spectrum cable connection that worked more or less reliably (except weekend evenings, or DNS anytime)–until virtual school started Monday, November 9, 2020. It hit like a ton of bricks when local middle and high schools went virtual very abruptly this week. Our Internet connection was largely unusable the first day, and not much better the second. I went through the normal diagnostic processes of rebooting the cable modem and router, but those things didn't help at all, so I started doing more detailed research on what I could do independent of Spectrum.
The short answer is that upgrading to a new DOCSYS 3.1 modem seems to have helped with reliability when the Spectrum network is under stress. If you are on Spectrum and your current modem is a DOCSYS 3.0 modem, try to get Spectrum to upgrade it to a DOCSYS 3.1 modem or go to the expense of doing it yourself. If you do it yourself, make sure to get one that Spectrum supports; see the rest of the article for more information.
Internet Service Provider Alternatives
I've been unhappy with the reliability of our Spectrum connection (my expectations were set high by several years in a neighborhood with Verizon FIOS service), but quickly learned that ATT does not offer DSL at our address and that I'm just over the hill for the line-of-sight necessary for a 186Networks fixed wireless connection. ViaSat, Hughes and the other fixed wireless vendor all looked to be slower, more expensive and less reliable. I knew that our T-mobile cell service was not a good short-term substitute, and a friend came by and confirmed that Verizon cell service wasn't any better at our house. Like or not, I'm stuck with Spectrum for the moment.
Existing Spectrum-supplied Cable Modem
Next, I looked up the specs on the Spectrum-supplied modem–a Cisco DCP 3216, which is DOCSYS 3.0 compliant and supports 16 download and 4 upload channels (16/4). Cisco describes this product as “retired” from support.
I started doing research on the DOCSYS 3.0 standard and the DOCSYS 3.1 standard, along with the differences between modems. I learned that DOCSYS 3.0 implements up to 32 download channels and 8 upload channels, but that the Spectrum-supplied modem only supported half of those channels. The DOCSYS 3.1 standard offers backward compatibility with 3.0, and adds an additional higher speed protocol. Some websites suggested that 32 channel modems perform better under network congestion than 16 channel modems. I decided that I needed a new modem and decided to pay for my own instead of trying to work with Spectrum support to get them to upgrade my modem. To reboot the Spectrum modem, I have to go to the basement to physically power cycle it. With my own device, I can log in from my tablet or computer and reboot it without having to go to the wiring closet to power cycle it.
Picking a New Modem
In researching modems, the first place that I looked was the Spectrum website to find which modems they support on their network. I looked primarily at the Netgear offerings because I know that the local Best Buy stocks some Netgear products. Although I don't currently have a plan that requires a DOCSYS 3.1 modem, I decided to look only at 3.1 compliant modems, even though a DOCSYS 3.0 modem with 32/8 channels would be much less expensive. I ended up deciding on a Netgear CM1000, as I could not see an easy way to take advantage of the channel bonding or higher speed Ethernet offered in the Netgear CM1100, CM12000 and CM2000 modems.
Spectrum supports the Netgear CM1000 for 1 gigabit service, which is a possible alternative if network reliability doesn't improve.
Our local Best Buy had a CM1000 in stock, so I bought it. There were also some less expensive Arris models in stock, but I didn't know the specific models that Spectrum supports.
Installing the Netgear CM1000
Installation was easy, but requires a cellphone data connection through a cell tower in order to tell Spectrum the MAC address of the new modem during the activation process. It only takes about 10 minutes, but allow an hour for fumbling around with finding your Spectrum user ID, password, the MAC address of the new router, and setting things up. The Spectrum instructions are reasonably good and worked once I got all of the necessary information.
With most devices, you have to update firmware, but this is not the case with cable modems; the carrier controls updating firmware. Spectrum appears to have updated my device to the most recent firmware.
Anecdotally, browsing response seems better. In looking at the Netgear CM1000 status page, it is clear that having 32 DOCSYS 3.0 channels is probably better under congestion than 16, as shown in Figure 1, but the real surprise is that the even though my service level–the lowest–does not require the DOCSYS 3.1 OFDM channels, Spectrum appears to use them if the modem supports them. Figure 2 shows the DOCSYS 3.1 channels and that the 3.1 channels are the only ones with errors; this suggests that when the DOCSYS 3.1 OFDM channels are available, Spectrum uses them. If most of the modems in my neighborhood are equivalent to the Cisco DPC3216, we will relieve pressure on those 16 channels and have better service due to a largely uncontested OFDM channel.
The approximately 25% correctable error rate shown is going to require more research to find out if there are wiring connection problems, or if this is a normal error rate.
I've previously had long-term DNS latency problems with Spectrum, and installed a local DNS (Pihole) on a Raspberry Pi B. This fixed the DNS latency problem and improved end-user response time significantly.
- Written by Bruce Moore
- Hits: 1153
Google Analytics Referral Spam
On March 17th, 2015, Google Analytics showed that my site got a big spike in referral traffic from a web site that didn't make sense as a legitimate referral site. I went to a domain name registration look-up site Whois.net and found that the domain had been registered the day before. More investigation showed that this was "referral spam" where no one ever looked at my site. The spammer spoofed the Google Analytics ID in a call to Google Analytics, so it showed up as a referral. In 2014, 2015 on up to about 2017, this was common, but Google Analytics has gotten much better at removing referral spam before you get to reports, so I've stopped reporting on it and have removed the various specific articles from my tech blog.
Trends in Referral Spam
It is hard to tell just how often this is occurring on any site other than one’s own, but Google Trends may offer some additional information, as shown in the dynamic figure below showing interest in the search term “referral spam”. Use the link https://trends.google.com/trends/explore?date=all&geo=US&q=referral%20spam. Unfortunately, you can't embed Google Trends data anymore.
Fixing the Problem
My first reaction in addressing referral spam was to add a line to .htaccess to block these spam referrals (see http://www.htaccess-guide.com/deny-visitors-by-referrer/ for a description of how to do this) but with more research, it turns out these referrals weren’t referrals to my site at all, but were insertions of fake referrals into my Google Analytics reports. As was the case with
darodar.com, the clear intent is to cause webmasters to go to an unfamiliar site when they see a reference in their Google Analytics reports. Whether the motivation is to generate traffic to their site or to cause webmasters to visit a site that will download malware is unknown.
Based upon the instructions in Removing Referral Spam from Google Analytics, I checked the hostname on the referrals, and all showed “(not set)”–a clear sign that no one ever touched my site and that these were inserted into Google Analytics to get me to click
social-buttons.com to generate traffic or download malware onto my computer.
Removing Referral Spam from Google Analytics provides a good description of the problem and some solutions. Understanding and eliminating referrer spam in Google Analytics gives another good description of referral spam and a programmatic solution that is appropriate for plug-in developers but not for administrators of WordPress, Joomla and other content management system (CMS) based sites.
An alternative is to switch to self-hosted Piwik for your web analytics; if you do this, it will be immediately clear that the vast majority of Google Analytics referral spam is of the spoofed variety rather than the crawler variety. Piwik does not have the advertising integration nor does it have the demographic information, but for many small-traffic sites it can provide much more information. See Using Piwik as an Alternative to Google Analytics on this web site for more information on why Piwik might work for you and how to implement it.
Useful Commands and Web Sites for Investigating Referrers
For investigating a referrer, here are some useful commands and web sites:
- TCPIPutils is a great site for looking up data on an domain or IP address
- For domain registrations, the command line
whois social-buttons.comis very convenient as is https://www.whois.net/
- For IP lookups,
dig social-buttons.comis convenient, as is http://ip-lookup.net/index.php
- Better Business Bureau
- To view a site in character mode so that malware doesn’t get downloaded, use
curl -L. These are commonly installed on Linux machines, but will require additional software on Windows and OS X, as discussed below.
- To look up a lot of information on an IP address in one place http://www.tcpiputils.com/browse/ip-address will give you a lot of information quickly.
Command Line Utilities
To use the
curl commands on Windows and OS X, you will need to install additional software:
- On Windows, install Cygwin and add the
- On OS X, install MacPorts and add the
Cygwin and MacPorts have many additional command line and graphical utilities that make life easier in Windows and OS X.
For more information on referral spam, see
- Social-buttons.com Referral Spam
- Best-seo-solution.com Referral Spam
- justprofit.xyz Referral Spam
- Get-free-social-traffic.com Referral Spam
- Video--production.com Referral Spam
- Rankscanner.com Referral Spam
- Success-seo.com Referral Spam
- Videos-for-your_business.com Referral Spam
- Semaltmedia.com Referral Spam
- 100dollars-seo.com Referral Spam
- Written by Bruce Moore
- Hits: 3177
WiFi Mesh with Wired Backbone
With the lockdown in Michigan for the pandemic in March of 2020, we had to start doing Zoom sessions from rooms where we had never needed good WiFi before, and discovered that our home wiFi network was not up to the task. I was using a four or five year old TPLink Archer C7, a somewhat older DLink DIR-860 running DD-WRT, and Trendnet box in the garage to upgrade maps in a car. It was a kludge of old equipment that was not up to the new task.
After reading several online sources I found a Wirecutter article on mesh routers which led me to an section way down at the bottom about using a wired backbone. I decided to go with a commercial solution from Ubiquity using a wired backbone. When our house was built, they still put in phone jacks, but used Cat 5e cable in a star configuration. When we had some wiring done when we moved in two years ago, I had them replace the RJ-11 plugs with RJ-45 plugs and central telephone punchdown block with an RJ-45 patch panel. This allowed me to use a wired backbone instead of a WiFi mesh.
I ordered three Ubiquity UniFi AP AC Lite access points for about $85 each, and installed the UniFi Network Management Controller on my laptop. I placed one in the basement office, one in the basement under the floor of the main living area, and one in a phone nook off the kitchen using the existing phone/RJ-45 plugs, and then provisioned them using the UniFi Network Management Controller. I had to use
scp to update firmware on one device, but I think if I waited a little longer, it would have automatically updated the firmware, as both of the others went fine. I used the provided power injection plugs instead of power over ethernet (POE), but will probably put a POE switch in the central cabinet at some point when I permanently mount the access points.
The only hiccup is that the POE power injection transformers for the UniFi devices does not play well with the Netgear PL1000 power line Ethernet devices that I had been using for a RoKu TV Streaming device because the WiFi signal in that location wasn't very good. With the improved WiFi signal, I don't need the power line Ethernet any more.
After six months, I have not had to reboot or otherwise touch any of the access points–something I've had to do on WiFi routers occasionally for every router I've ever had. We have one SSID, so you never have to manually switch networks as was needed with the old equipment. The speed is great, and the coverage is great too, even for updating the maps in the car in the garage.
This is a slightly technical solution, but if you have or can install a wired backbone, it is great.
- Written by Bruce Moore
- Hits: 1134
Canon T4i and T5i Work as Webcams with Canon Beta Driver
With the pandemic, everyone went to online meetings, and suddenly, built-in webcams just didn't look so good. Just about all models of good, reliable webcams were sold out and on back order by mid-March, so I started looking for ways to use my Canon and Olympus cameras. Canon quickly released a beta for a webcam utility. Although Canon does not officially support them, I've been able get my Canon T4i and T5i to work as webcams using the Canon EOS Webcam Utility. The G5X Mark II is supported, but the G5X (Mark I) is not, and I have not been able to get my G5X (Mark I) to work.
Battery life on the T4i and T5i is about 1.5 hours--enough for most Zoom calls, but you will always want to have a spare battery on the charger.
The picture on Zoom is has a black box around it, but I have not tried all of the settings yet, so there may be a way to fix this. In any case, it is a much better picture than the 15-year old 320px webcam that I've been using or the built-in one on my Lenovo W540.
- Written by Bruce Moore
- Hits: 1832