Home Automation on William Floyd (low-quality images)

Extracting Energy Data from Home Assistant

William Floyd — Thu, 09 Apr 2026 00:00:00 +0000

Couldn’t find a complete guide for this.

Install https://github.com/klausj1/homeassistant-statistics

Run like so:

Using this as the timestamp string: %Y-%m-%dT%H:%M:00Z

Once downloaded, to find what id you should filter to (you might already know the entity name from Home Assistant):

cat export_counters.json |
    jq 'map(.id | select(contains("consumption")))'

In my case, the file is large, so it’s best to pre-process the file and filter it to just the target entities values:

cat export_counters.json |
    jq 'map(select(.id=="opower:xxx_energy_consumption")) | first | .values' > export_filtered_counters.json

Now you have your power data, ready to process!

This is where things can get as crazy as you like, depending on your affinity for JQ.

Simple Route (CSV / Excel)

If you want to do stuff in Excel (or any similar product) you might want to further process this data:

cat export_filtered_counters.json |
    jq 'map(.datetime |= (fromdate | strftime("%Y-%m-%d %H:%M:00")))' |
    yq -o csv > export_filtered_counters.csv

This gets you a CSV with date strings that Excel can work with:

datetime,sum,state,delta
2025-04-05 00:00:00,3010.138800000007,1.041,0.0
2025-04-05 01:00:00,3010.948800000007,0.81,0.8099999999999454
2025-04-05 02:00:00,3011.637600000007,0.6888,0.688799999999901
2025-04-05 03:00:00,3012.369600000007,0.732,0.7319999999999709
2025-04-05 04:00:00,3013.131600000007,0.762,0.762000000000171
2025-04-05 05:00:00,3013.871400000007,0.7398,0.7397999999998319

You’ll probably just want the state and datetime column.

Draw the Rest of the Owl

I get carried away, and built out JQ functions to model my utility rate options. Ultimate flexibility, much easier than wrangling in Excel.

Script. Usage is like so:

cat export_filtered_counters.json | jq -f calculate.jq

(
    [
        153.90,
        118.72,
        84.34,
        80.71,
        101.59,
        116.60,
        136.78,
        192.62,
        207.16,
        144.40,
        135.87,
        114.62
    ]
    | add
) as $actual |
def isHoliday($dateTime):
    ($dateTime | strftime("%m") | tonumber) as $month |
    ($dateTime | strftime("%d") | tonumber) as $day |
    ($dateTime | strftime("%w") | tonumber) as $weekday |
    # New Year's Day: January 1
    ($month == 1 and $day == 1) or
    # Memorial Day: last Monday in May
    ($month == 5 and $weekday == 1 and $day >= 25) or
    # Independence Day: July 4
    ($month == 7 and $day == 4) or
    # Labor Day: first Monday in September
    ($month == 9 and $weekday == 1 and $day <= 7) or
    # Thanksgiving Day: fourth Thursday in November
    ($month == 11 and $weekday == 4 and $day >= 22 and $day <= 28) or
    # Christmas Day: December 25
    ($month == 12 and $day == 25)
    ;
# Examples
# RESIDENTIAL GENERAL USE AND SPACE HEAT - ONE METER: 2RS6A
def _residential_general_use_and_space_heat_one_meter_2rs6a:
    . as $dot |
    (
        group_by(.month) |
        map(
            (first | .month) as $groupMonth |
            {
                month: $groupMonth,
                state: (map(.state) | add)
            }
        )
    ) as $monthstates |
    $dot | map(
        . as $entry |
        ($monthstates | map(select(.month==$entry.month)) | first | .state) as $thisMonthstate |
        $entry |
        (.month >=6 and .month <= 9) as $isSummer |
        if $isSummer then
            0.10021
        else # Winter
            # That is, the rate is 0.06966 for the first 1000, and 0.06123 for everything past 1000
            if ($thisMonthstate>1000) then
                # Calculate the average charge given the 1000 split
                ((($thisMonthstate-1000) * 0.06123) + (1000 * 0.06966)) / $thisMonthstate
            else
                0.06966
            end
        end |
        $entry + {rate: .}
    )
    ;
# RESIDENTIAL GENERAL USE – ONE METER: 2RS1A
def _residential_general_use_one_meter_2rs1a:
    map(
        . as $entry |
        (.month >=6 and .month <= 9) as $isSummer |
        if $isSummer then
            0.10021
        else # Winter
            0.07735
        end |
        $entry + {rate: .}
    )
    ;
# 2RTOU, With Net Metering 2RTOUN
def _2rtou_with_net_metering_2rtoun:
    map(
        . as $entry |
        (.month >=6 and .month <= 9) as $isSummer |
        (
            (.weekday > 0 and .weekday < 6) # Weekday
            and (.hour >= 16 and .hour < 20) # 4pm-8pm
            and (isHoliday(.datetime) | not) # Not holidays
        ) as $isOnPeak |
        (.hour >= 0 and .hour < 6) as $isSuperOffPeak |
        if $isOnPeak then
            if $isSummer then
                0.26838
            else
                0.20151
            end
        elif $isSuperOffPeak then
            if $isSummer then
                0.03834
            else
                0.02879
            end
        else # Off-Peak
            if $isSummer then
                0.07668
            else
                0.05758
            end
        end |
        $entry + {rate: .}
    )
    ;
def _rtou2_with_net_metering_rtou2n:
    map(
        . as $entry |
        (.month >=6 and .month <= 9) as $isSummer |
        (
            (.weekday > 0 and .weekday < 6) # Weekday
            and (.hour >= 16 and .hour < 20) # 4pm-8pm
            and (isHoliday(.datetime) | not) # Not holidays
        ) as $isOnPeak |
        (.hour >= 0 and .hour < 6) as $isSuperOffPeak |
        if $isSummer then
            if $isOnPeak then
                0.2725
            else # Off-Peak
                0.0681
            end
        else # Winter
            if $isSuperOffPeak then
                0.0397
            else # Off-Peak
                0.0793
            end
        end |
        $entry + {rate: .}
    )
    ;
def _apply_rate:
    map(
        . + { cost: (.state*.rate)}
    );
# Monthly Costs
def _add_monthly_cost:
    . + 14.25*12;
def _add_rate_fees:
    .
    # | map(.rate *= 1.287) # TAX ADJUSTMENT
    | map(.rate += 0.00103) # PROPERTY TAX SURCHARGE
    | map(.rate += 0.01084) # TRANSMISSION DELIVERY CHARGE
    ;
def _cost_adder:
    map(.cost) | add;
def _analyze_by($grouperField; analyzer):
    group_by(.[$grouperField]) |
    map(
        (first | .[$grouperField]) as $group |
        {
            $grouperField: $group,
            data: analyzer,
            # total_cost: _cost_adder
        }
    );
def _flatten_analyze($obj):
    map(
        . as $entry
        | if (($entry.data | type) == "array") then
            $entry.data | _flatten_analyze($entry | del(.data))
        else
            $entry + $obj
        end
    ) | flatten;
def _flatten_analyze:
    _flatten_analyze({});
def _is_outlier($state):
    # Heuristic
    $state > 6 or $state < 0;
def _fix_outliers:
    (
        map(select(_is_outlier(.state) | not))
    ) as $withoutOutliers |
    (
        $withoutOutliers
        | group_by(.hour)
        | map({ (first | .hour | tostring): (map(.state) | add / length) })
        | add
    ) as $hourAvgs |
    map(
        if _is_outlier(.state) then
            . + { state: $hourAvgs[.hour | tostring] }
        else
            .
        end
    )
    ;
(0.550) as $load | # kWh
(0.920) as $RTE | # Round Trip Efficiency
(0.850) as $SoCWindow | # State of Charge window, eg 10% to 90% = %80
2.875 as $hours | # Intended duration of battery runtime
($hours*($load/$RTE)/$SoCWindow) as $batterySize |
def _adjust_usage:
    map(
        if (.hour >= 16 and .hour < (16+$hours)) then
            .state -= $load
        elif (.hour >= 0 and .hour < ($hours)) then
            .state += ($load/$RTE)
        else
            .
        end
    )
    ;
map(
    . as $dot
    | .datetime | fromdate
    | (strftime("%H") | tonumber) as $hour # 0 through 23 |
    | (strftime("%w") | tonumber) as $weekday # 0 is Sunday, 6 is Saturday |
    | (strftime("%m") | tonumber) as $month # 1 through 12 |
    |
    {
        datetime: .,
        state: $dot.state,
        hour: $hour,
        weekday: $weekday,
        month: $month
    }
)

# | _fix_outliers

# | _adjust_usage

| _residential_general_use_one_meter_2rs1a # RESIDENTIAL GENERAL USE – ONE METER: 2RS1A
# | _residential_general_use_and_space_heat_one_meter_2rs6a # RESIDENTIAL GENERAL USE AND SPACE HEAT - ONE METER: 2RS6A 
# | _2rtou_with_net_metering_2rtoun # 2RTOU, With Net Metering 2RTOUN
# | _rtou2_with_net_metering_rtou2n # RESIDENTIAL TIME OF USE TWO-PERIOD

| _add_rate_fees
| _apply_rate

# | _analyze_by("month";(_analyze_by("hour";_cost_adder))) | _flatten_analyze

| map(.cost) | add | _add_monthly_cost as $calculated
| {
    actual: $actual,
    calculated: $calculated
}

# | (
#     (
#         _2rtou_with_net_metering_2rtoun
#         | _add_rate_fees
#         | _apply_rate
#         | map(.cost) | add | _add_monthly_cost
#     ) -
#     (
#         _adjust_usage
#         | _2rtou_with_net_metering_2rtoun
#         | _add_rate_fees
#         | _apply_rate
#         | map(.cost) | add | _add_monthly_cost
#     )
# )
# | {"Yearly Savings ($)": ., "Battery Size (kWh)": $batterySize}

Home Automation at School

William Floyd — Wed, 21 Apr 2021 00:00:00 +0000

I love home automation - I’ve spent far longer writing automation routines and hacking together my own devices and programs than I have ever saved by doing so, and that’s perfectly fine. One unique aspect of my setup, however, is that I cannot control the network I must use - in my Uni dorm, I am not allowed to run my own router, and so all IoT devices must connect to the school wireless network. There abound dozens of Google Home devices, Chromecasts, and so on, all accessible on the same network - but not from the wired connection that my server/desktop uses.

Here then is my solution: MQTT everything I can. From my lights, to light sensor, to coffee maker to desktop software, I bounce it through a MQTT server hosted on a VPS. I use Home Assistant, so automatic discovery is easy on most things, especially ESPHome. In fact, I disabled direct Home Assistant connectivity entirely on these devices, which works well enough for me to live with.

But how to flash?

Given that my Home Assistant instance isn’t even on the same network as the IoT devices, how do I update the firmware? Using my laptop, I can connect to the wireless network that they reside on, and using IP address sensors reported by these devices, flash them directly without needing local discovery. In fact, I can easily automate this for myself using a couple scripts and a minimal number of hard-coded values:

hassio.sh

#!/bin/bash

export HASS_SERVER=https://<server_url>:443
export HASS_TOKEN='<HA_Token>'

hass-cli ${@}

exit

lamps.sh

#!/bin/bash

declare -A aa

aa["gosund_lb1_1.yaml"]="sensor.desk_lamp_ip_address"
aa["gosund_lb1_2.yaml"]="sensor.bed_lamp_ip_address"
aa["gosund_lb1_3.yaml"]="sensor.floor_lamp_girlfriend_ip_address"
aa["gosund_lb1_4.yaml"]="sensor.desk_lamp_girlfriend_ip_address"
aa["gosund_lb1_5.yaml"]="sensor.floor_lamp_ip_address"

__flash() {

    __config="${1}"
    __entity_name="${aa[${__config}]}"
    echo "Getting ${__config} IP..."

    __ip="$(
        ./hassio.sh -o yaml state get \
            "${__entity_name}" |
            grep -E '^ *state' | sed -e 's/.* //'
    )"

    echo "IP: ${__ip}"

    if [ "${__ip}" == 'unavailable' ]; then
        echo 'Ignoring...'
    else
        echo "Flashing..."
        ./esphome.sh "${__config}" run --upload-port="${__ip}"
    fi
    
    echo

}

if [ "${#}" -gt 0 ]; then
    until [ "${#}" == 0 ]; do
        __flash "${1}"
        shift
    done
else
    for __config in ${!aa[@]}; do
        __flash "${__config}"
    done
fi

exit

this allows me to mostly painlessly flash my devices, though truth be told there is little need.

Custom software

I developed for myself a tool in Golang to help tie more of my devices together. It is rather uncreatively/cryptically named ha-mqtt-iot - that is, “Home Assistant MQTT Internet of Things”. I may rename this some day, but why bother. It is similar to IOTLink (which is Windows only), and HASS Workstation Service - they are great projects, but this one is mine, even if it is poorly written.

The gist of the software is that most (all?) device types supported by Home Assistant may be implemented using a selection of user defined commands. The most prominent example in my case is to enable/disable dark mode on my desktop. I automate this according to ambient light in my room, to better match the aesthetic I want. Additionally, I can use it to turn my desktop monitor off, without resorting to using a relay outlet, and even change the color temperature of my system. The script I use for this looks like the following:

#!/bin/bash

__monitor_i2c='dev:/dev/i2c-3'
__monitor_dpms='0xd6'
__monitor_brightness='0x10'
__monitor_standby='4'
__monitor_off='5'
__monitor_on='1'

__unknown() {
    echo "Unknown ${1}"
}

f2i() {
    awk 'BEGIN{for (i=1; i<ARGC;i++)
   printf "%.0f\n", ARGV[i]}' "$@"
}

com="${1}"
arg="${2}"

case "${com}" in
"command")
    case "${arg}" in
    "ON")
        xset dpms force on
        ;;
    "OFF")
        (
            #xset dpms force off
            #sleep 0.5s
            #ddccontrol -r "${__monitor_dpms}" -w "${__monitor_standby}" "${__monitor_i2c}" -f
            #sleep 2s
            ddccontrol -r "${__monitor_dpms}" -w "${__monitor_off}" "${__monitor_i2c}" -f
        ) &
        ;;
    *)
        __unknown "${arg}"
        ;;
    esac
    ;;
"command-state")
    echo -n "$(xset q | grep 'Monitor is' | sed -e 's/.* //' | tr '[:lower:]' '[:upper:]')"
    ;;
"color-temp")
    v="$(f2i "$(bc -l <<<"1000000/${arg}")")"
    ./scripts/run-in-user-session.sh gsettings set org.gnome.settings-daemon.plugins.color night-light-temperature "${v}"
    ;;
"color-temp-state")
    v="$(./scripts/run-in-user-session.sh gsettings get org.gnome.settings-daemon.plugins.color night-light-temperature)"
    echo -n "$(f2i "$(bc -l <<<"1000000/${v/* /}")")"
    ;;
"brightness")
    ddccontrol -r "${__monitor_brightness}" "${__monitor_i2c}" -w "${arg}"
    ;;
"brightness-state")
    echo -n "$(ddccontrol 2>/dev/null -r "${__monitor_brightness}" "${__monitor_i2c}" | tail -n 1 | grep -o '/[0-9]*/100' | sed -e 's|^/||' -e 's|/.*||')"
    ;;

*)
    __unknown "root command ${com}"
    ;;
esac

exit

Note that the display doesn’t respond to being turned back on, so this is somewhat incomplete, but it’s good enough for my needs. The corresponding portion of the config for ha-mqtt-iot looks like the following:

"lights": [
    {
        "info": {
            "name": "Desktop Monitor",
            "id": "monitor"
        },
        "command": [
            "./scripts/monitor.sh",
            "command"
        ],
        "command_state": [
            "./scripts/monitor.sh",
            "command-state"
        ],
        "command_color_temp": [
            "./scripts/monitor.sh",
            "color-temp"
        ],
        "command_color_temp_state": [
            "./scripts/monitor.sh",
            "color-temp-state"
        ],
        "command_brightness": [
            "./scripts/monitor.sh",
            "brightness"
        ],
        "command_brightness_state": [
            "./scripts/monitor.sh",
            "brightness-state"
        ],
        "brightness_scale": 100,
        "update_interval": 5
    }
]

Pretty simple. This makes custom system sensors trivial. For example, to show my system IP, I use the following:

"sensors": [
    {
        "info": {
            "name": "IP Address Desktop Solus",
            "id": "ip-address-desktop-solus",
            "icon": "mdi:ip-network"
        },
        "command_state": [
            "/bin/bash",
            "-c",
            "ip -j address show eno1 | jq -r '.[0].addr_info[0].local'"
        ],
        "update_interval": 10
    }
]

Some common use cases are built in as well. Currently, this includes laptop displays (as lights) and batteries (as sensors), as well as Crypto prices (through the CoinGecko Golang library). These are really easy to call. An exhaustive example is quite short:

"builtin": {
    "prefix": "Name Prefix ",
    "backlight": {
        "enable": true,
        "temperature": false,
        "range": {
            "minimum": 0.025,
            "maximum": 0.95
        }
    },
    "battery": {
        "enable": true
    },
    "crypto": [
        {
            "coin_name": "dogecoin",
            "currency_name": "usd",
            "update_interval": 1,
            "icon": "mdi:currency-usd"
        }
    ]
}

This lets me tailor my setup to each machine I’m using, while still enjoying the benefits of Home Assistant MQTT Discovery. The primary limitation at present is the inability to signal to ha-mqtt-iot from another process - it can only poll for changes. This will be addressed one day, when it is important for my own needs.

How to host?

But the question is now, how do I access my HomeAssistant instance if it’s also hosted at school? I most certainly don’t have a public IP, so in comes AutoSSH. I’m not sure which is the best one at this stage, but refer to this repo and check the various forks of the parent project.

I have configured on my VPS a docker image that accepts reverse SSH tunnelling, authorized only to the key of the HA addon. From my docker-compose.yml:

homeassistant:
    image: "docker.io/panubo/sshd"
    container_name: homeassistant
    environment:
        - TCP_FORWARDING=true
        - GATEWAY_PORTS=true
        - SSH_ENABLE_ROOT=true
        - DISABLE_SFTP=true
    volumes:
        - "./hassio/authorized_keys:/root/.ssh/authorized_keys:ro"
        - ./docker-config/hassio/data/:/data
        - ./docker-config/hassio/keys/:/etc/ssh/keys
    ports:
        - "<MY_PORT>:22"
    restart: unless-stopped
    hostname: "homeassistant"

This is then reverse proxied to using Caddy, to expose the website on a subdomain of a website. From my Caddyfile:

<MY_SUBDOMAIN>.{$MY_DOMAIN} {
    reverse_proxy homeassistant:8123
    encode gzip
}

Pretty simple, but not without some hiccups now and again - I occasionally have to restart the sshd docker on my VPS if something goes wrong with HomeAssistant.

Bonus: Android tie in

I use Sleep As Android to track my sleeping patterns, and as my alarm clock. Using Tasker, I can run an action when I begin sleep tracking, which (using a HomeAssistant plugin for Tasker) can call a script on my HomeAssistant instance to turn off my lights (only if I’m home, of course). Similarly, it turns on my bedhead light when my alarm goes off in the morning, and could optionally make me coffee…

DIY 'Smart' Coffee Maker

William Floyd — Sun, 05 Jul 2020 00:00:00 +0000

I have recently (last couple months) been getting into home automation, specifically using Home Assistant. I also wanted to get a coffee maker that used K-Cups. Thus the goal was born - to get a coffee maker working with Home Assistant, such that I could automate it to fit my morning routine.

I first searched for new coffee makers, looking to see what the easiest ones would be to hack into and automate. I got so far as adding a sleek new brand name model to my Amazon cart, but thankfully reason and frugality prevailed! Instead, I opted to swing by the local Goodwill the next day, to see what secondhand options might be available. As luck would have it, I found a no-frills, single button coffee maker for the princely sum of $4, and I was on my merry way.

Later that evening, opening the black box up (thankfully I had the correct triangular screwdriver head), I found very easily hackable innards.

The control board (front, back) is very simple - two buttons in parallel, an LED, and a couple resistors. I’m not quite sure what the logic is upstream (it un/latches on each press), but the buttons short to ground, and this is something I can work with. The control board has wires for ground, signal, and 5V, in that order.

The ordeal of hacking into the thing, and my missteps trying to do so, is rather boring really, but it produced this:

The brains I added was a Wemos D1 Mini (thanks, Aliexpress!), with a 3.3V<->5V converter between to sense when the coffee maker is on, and also turn on a mosfet to short the button to ground. I initially had issues with the coffee maker turning on whenever I plugged the D1 Mini in, but a pulldown resistor solved this.

I also opted for an external USB plug - this allows me to possibly reflash this without opening the enclosure, and also meant I did not need to splice into the mains line inside, which I was reluctant to do.

I initially flashed and programmed the coffee maker with Tasmota, but almost immediately opted to use ESPHome instead. I like Tasmota and ESPHome, I was just able to get more fine control of the setup using ESPHome in this instance, especially concerning the button logic.

The (important) code is as follows:

binary_sensor:
  - platform: gpio
    device_class: power
    internal: true
    id: power_on
    pin: D2
    
switch:
  - platform: gpio
    pin: D1
    id: relay
    restore_mode: ALWAYS_OFF
  - platform: template
    name: "Coffee Maker"
    icon: "mdi:coffee-maker"
    lambda: |-
      if (id(power_on).state) {
        return true;
      } else {
        return false;
      }
    turn_on_action:
    - switch.turn_on: relay
    - delay: 50ms
    - switch.turn_off: relay
    turn_off_action:
    - switch.turn_on: relay
    - delay: 50ms
    - switch.turn_off: relay

The beauty of the way in which I can detect signals and simulate a button press is that ESPHome senses when I use the physical button, and updates accordingly. I can use the coffee maker absolutely as normal, the ‘Smart’ part of it is optional.

The rest of my setup using this coffee maker is somewhat outside the scope of this post, but it includes using Tasker on my phone to run a ‘Wakeup’ automation on Home Assistant, which will - along with turning on my bed-head lamp, and switching my monitor on before putting it to sleep - check if I turned on a ‘Morning Coffee’ user input. If I remembered to prep my coffee the night before, and turned on the switch, I will wake up to a hot cup of coffee (or, the sound of a coffee-in-progress). I can also command Alexa and Google Assistant to do my bidding and make me a coffee. At least someone does what I want…

Truly, we are living in the future.