Trimming the Fat - Image Optimization
Since I started this site I have simply been using a standard PNG imnage for all of the visual assets on the site. The includes the card image, an image on a post, the watches, literally every thing. All of the images have been sized to an appropriate resolution, however, this still isn't enough. I will not need high resolution images with the current functionality of the site. At some point, I plan to add a gallery or just host a pixelfed instance again, but, that is still to be decided. My entire point is by converting the images to WebP I can provide viewers with a better experience with what the site offers.
These are the high level steps I am taking to migrate the images on the site:
- (Providing Site Context)
- Convert all the PNGs to WebP
- Identify PNG References
- Change References to WebP
- Test and Verify
As a note, and to echo my opinons on AI this is an excellent project that can be accelerated using the assistance of self hosted models. If you do use AI, be sure to understand the fundamentals and not blindly run anything. Thank you for listening!
(Providing Site Context)
How did we get into this situation?
Well... seanland has been an evolution of custom static site generators. I have built a few personal sites. I hope at some point to go through all my old code and make a post of the evolution to today's version. With this evolution, the code, structure and practices has been based on what I wanted to do at the time. One thing I wanted to do was use PNGs (this is a progression from weighty JPGs back in the day). Now, when I look at some of these pages - even after shrinking them down - they are just too heavy. The same page we are going to use as a benchmark is image heavy. It loads 3.75 MB of images. That's too much. Yes, I could lazy load, but, I don't really want to. I want a crisp, basic site.
Since, this is a custom static site generator I know that all the images are store in one high level directory. It does contain sub-directories, so any conversion will also have to be done recursively. Another note, is that the logo is SVG (this will not be converted), the UNESCO Tracker Key images are 80px by 80px JPGs that we will want to take into account during the conversion. Those images are also stored in this "high level" directory.
Beyond those notes, we will have this backed up through our different git branches. If something goes wrong, we just blow away the branch and restart.
Convert all the PNGs to WebP
This is where I am going to use gemma4:31b to help create the script. Gemma4 was selected as the model of choice, simply becuase it is currently my favourite all purpose model.
create a python script that does the following:
* select a directory and identify images that are jpg, jpeg and png
* list the size of the file in the output
* convert the file to a webp image
* list the size of the web image
* compress the image and retain a high quality image.
After 39 seconds of processing on my 395+ via ollama and Open WebUI this is the unedited full output (minus the removal of code blocks for display purposes).
To create this script, you will need the **Pillow** library, which is the standard Python library for image processing.
### Prerequisites
First, install the Pillow library via terminal or command prompt:
pip install Pillow
### The Python Script
import os
from tkinter import Tk, filedialog
from PIL import Image
def get_file_size(file_path):
"""Returns the file size in Kilobytes."""
return os.path.getsize(file_path) / 1024
def convert_to_webp():
# 1. Select a directory
root = Tk()
root.withdraw() # Hide the main tkinter window
folder_selected = filedialog.askdirectory(title="Select Folder Containing Images")
if not folder_selected:
print("No folder selected. Exiting...")
return
# Define supported extensions
extensions = ('.jpg', '.jpeg', '.png')
# Identify images in the selected directory
files = [f for f in os.listdir(folder_selected) if f.lower().endswith(extensions)]
if not files:
print("No matching images found in the selected directory.")
return
print(f"Found {len(files)} images. Starting conversion...\n")
print(f"{'Filename':<30} | {'Original (KB)':<15} | {'WebP (KB)':<15} | {'Reduction'}")
print("-" * 75)
for filename in files:
# Setup paths
img_path = os.path.join(folder_selected, filename)
file_name_no_ext = os.path.splitext(filename)[0]
webp_path = os.path.join(folder_selected, f"{file_name_no_ext}.webp")
# 2. List original file size
original_size = get_file_size(img_path)
try:
# 3. Convert the file to webp
with Image.open(img_path) as img:
# 5. Compress and retain high quality
# quality=80 is generally the "sweet spot" for high quality vs size.
# Use 90+ for nearly lossless quality.
img.save(webp_path, "WEBP", quality=80, method=6)
# 4. List the size of the webp image
webp_size = get_file_size(webp_path)
reduction = ((original_size - webp_size) / original_size) * 100
print(f"{filename:<30} | {original_size:>13.2f} | {webp_size:>13.2f} | {reduction:>8.1f}%")
except Exception as e:
print(f"Error processing {filename}: {e}")
print("\nConversion complete!")
if __name__ == "__main__":
convert_to_webp()
### How it works:
1. **Directory Selection**: I used `tkinter.filedialog` so you don't have to hard-code a folder path; a window will pop up asking you to pick a folder.
2. **Filtering**: It uses a tuple `('.jpg', '.jpeg', '.png')` and `.lower().endswith()` to ensure it catches all variations of the requested formats.
3. **Size Calculation**: The `get_file_size` helper function converts the raw bytes provided by `os.path.getsize` into Kilobytes (KB) for better readability.
4. **Conversion & Compression**:
* **`quality=80`**: This is the primary compression setting. WebP quality ranges from 0 to 100. `80` provides a significant reduction in size while remaining visually indistinguishable from the original for most users.
* **`method=6`**: This tells the WebP encoder to use the slowest (and most efficient) compression method to get the smallest possible file size for that specific quality level.
5. **Output**: The script prints a formatted table showing the original size, the new size, and the percentage of space saved.
And... it created a script that requires an extra library strictly for a pop up window. Welcome to AI. This is it's usage in a nutshell and this sums up the good and bad of it. What do we do next? Well, we can add the library (no, thank you), we can go down a prompt spiral of "fix this" or "change that", or we can just change this ourselves.
Also, since I lacked the appropriate context. It never processed beyond the root directory. I want it to process recursively. It also never removed the original file. I only wanted to retain the conversion. Points I never explicitly called out in the original prompt. You get the point.
Fast forward, below is the script I am using:
import os
import argparse
from PIL import Image
def get_file_size(file_path):
"""Returns the file size in Kilobytes."""
return os.path.getsize(file_path) / 1024
def process_images(top_level_dir):
# Check if directory exists
if not os.path.isdir(top_level_dir):
print(f"Error: The path '{top_level_dir}' is not a valid directory.")
return
# Define supported extensions
extensions = ('.jpg', '.jpeg', '.png')
# We'll store a list of all images to process first to count them
# or just process them on the fly. Let's do it on the fly for memory efficiency.
print(f"Scanning directory tree: {top_level_dir}")
print(f"{'Relative Path':<50} | {'Original (KB)':<15} | {'WebP (KB)':<15} | {'Reduction'}")
print("-" * 95)
processed_count = 0
# os.walk yields a 3-tuple: (current_folder_path, subdirectories, files_in_current_folder)
for root, dirs, files in os.walk(top_level_dir):
for filename in files:
if filename.lower().endswith(extensions):
# Construct full paths
img_path = os.path.join(root, filename)
file_name_no_ext = os.path.splitext(filename)[0]
webp_path = os.path.join(root, f"{file_name_no_ext}.webp")
# Calculate relative path for cleaner output in the terminal
rel_path = os.path.relpath(img_path, top_level_dir)
original_size = get_file_size(img_path)
try:
with Image.open(img_path) as img:
# quality=80: high quality, method=6: best compression efficiency
img.save(webp_path, "WEBP", quality=80, method=6)
webp_size = get_file_size(webp_path)
reduction = ((original_size - webp_size) / original_size) * 100
processed_count += 1
print(f"{rel_path:<50} | {original_size:>13.2f} | {webp_size:>13.2f} | {reduction:>8.1f}%")
except Exception as e:
print(f"Error processing {rel_path}: {e}")
if processed_count == 0:
print("No matching images found in the directory tree.")
else:
print(f"\nConversion complete! Processed {processed_count} images.")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Recursively convert JPG/PNG images to high-quality WebP format.")
parser.add_argument("directory", help="Path to the top-level directory containing images")
args = parser.parse_args()
process_images(args.directory)
With three prompts and after waiting 133 (39 + 44 + 50) seconds I received the above code. That's the magic people see. The output of the script looks like this:
Scanning directory tree: images/
Relative Path | Original (KB) | WebP (KB) | Reduction
-----------------------------------------------------------------------------------------------
profile.png | 450.60 | 37.88 | 91.6%
self-hosted.png | 111.91 | 16.44 | 85.3%
...
2020/security.png | 88.97 | 20.64 | 76.8%
2020/words.png | 68.63 | 22.24 | 67.6%
2020/IMG_20201202_233023.jpg | 1677.69 | 1081.38 | 35.5%
2026/self-hosted-search.png | 37.85 | 13.00 | 65.6%
2026/bazzite-win4.png | 295.58 | 28.09 | 90.5%
Conversion complete! Processed 1266 images.
You can see as a small sample set the space savings some of the images have.
Before we move on let's remove - git remove - all the original files. We will run this script.
#!/bin/bash
# Set the target directory (defaults to current directory if no argument is provided)
TARGET_DIR="${1:-.}"
echo "Searching for .jpg, .jpeg, and .png images in Git index: $TARGET_DIR"
echo "------------------------------------------------------------"
# 1. Identify the files
# We use -print0 and read -d '' to safely handle filenames that contain spaces
FILES=$(find "$TARGET_DIR" -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" \))
# Check if any files were found
if [ -z "$FILES" ]; then
echo "No matching images found."
exit 0
fi
# List the identified files
echo "The following files will be removed from Git and Disk:"
echo "$FILES"
echo "------------------------------------------------------------"
# 2. Request confirmation
read -p "Are you sure you want to 'git rm' these files? (y/N): " CONFIRM
if [[ "$CONFIRM" =~ ^[Yy]$ ]]; then
# 3. Execute git rm
# -exec git rm {} + allows find to pass multiple files to one git rm command,
# which is much faster than running git rm for every single file.
find "$TARGET_DIR" -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" \) -exec git rm {} +
echo "Files have been removed and staged for commit."
echo "Run 'git commit -m \"Remove image assets\"' to finalize the change."
else
echo "Operation cancelled. No files were removed."
fi
I do really enjoy two things about AI generated scripts. They tend to have comments and processing outputs. Two things, I was skipped over when writing up "quick and dirty" scripts.
This is the output.
Searching for .jpg, .jpeg, and .png images in Git index: images/
------------------------------------------------------------
The following files will be removed from Git and Disk:
images/sites/site_1614.jpg
images/sites/site_1377.jpg
...
images/2026/bazzite-win4.png
images/default.png
images/meshtastic.png
images/chatsnld.png
------------------------------------------------------------
Are you sure you want to 'git rm' these files? (y/N):
After confirming you get a nice little output to finalize the changes.
...
rm 'images/IMG_20250919_190556_240_240.png'
Files have been removed and staged for commit.
Run 'git commit -m "Remove image assets"' to finalize the change.
Time to find all the places I need to change out the image files.
Identify PNG References
Fortunately, in the four python scripts that make up the static site generator, I only have six places where images or image extensions are hardcoded.
- Filter to copy watch images from repo to generated site
- Identifying watch images to use for the carousel
- Hardcode avatar in the footer
- File type for RSS enclosure URL
- The UNESCO Tracker had 'jpg' as an extension hardcoded
- The JSON visits have a reference to a hardcoded image.
The remainder of all the images will be contained in the webpages which are all generated from markdown files.
This is the script I used to uncover the images in the markdown files.
import os
import re
from pathlib import Path
import argparse
def find_images_in_file(file_path):
"""
Scans a file for Markdown and HTML image patterns.
"""
# Regular expressions for:
# 1. Markdown: 
# 2. HTML: <img src="url" ...> or <img src='url' ...>
md_pattern = r'!\[.*?\]\((.*?)\)'
html_pattern = r'<img[^>]+src=["\']([^"\']+)["\'][^>]*>'
images_found = []
try:
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
# Find all Markdown images
md_matches = re.findall(md_pattern, content)
for match in md_matches:
images_found.append({'type': 'Markdown', 'url': match})
# Find all HTML images
html_matches = re.findall(html_pattern, content)
for match in html_matches:
images_found.append({'type': 'HTML', 'url': match})
except Exception as e:
print(f"Error reading {file_path}: {e}")
return images_found
def scan_directory(root_dir):
"""
Recursively searches for .md files and identifies images within them.
"""
root_path = Path(root_dir)
if not root_path.is_dir():
print(f"Error: {root_dir} is not a valid directory.")
return
summary = {}
total_images = 0
# rglob('* .md') finds all .md files in current and sub-directories
for md_file in root_path.rglob('*.md'):
images = find_images_in_file(md_file)
if images:
summary[str(md_file)] = images
total_images += len(images)
return summary, total_images
def print_summary(summary, total_count):
"""
Formats and prints the final discovery report.
"""
print("\n" + "="*60)
print(f"IMAGE DISCOVERY SUMMARY")
print("="*60)
if not summary:
print("No images found in any markdown files.")
return
for file, images in summary.items():
print(f"\nš File: {file}")
for img in images:
print(f" - [{img['type']}] {img['url']}")
print("\n" + "="*60)
print(f"Total Images Found: {total_count}")
print(f"Total Files Containing Images: {len(summary)}")
print("="*60 + "\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Identify images in Markdown files recursively.")
parser.add_argument("directory", nargs="?", default=".", help="The directory to scan (defaults to current directory)")
args = parser.parse_args()
print(f"Scanning directory: {os.path.abspath(args.directory)}...")
results, count = scan_directory(args.directory)
print_summary(results, count)
The output looks like this:
Scanning directory: /images...
============================================================
IMAGE DISCOVERY SUMMARY
============================================================
š File: ../../README.md
- [HTML] __GHOST_URL__ /images/2021/04/IMG_20210411_141236.jpg
š File: ../../posts/2024-04-11-revisting-a-vacationing-mind.md
- [HTML] /images/2024/vacation-mexico-notes-2023_1200ma_.png
...
š File: ../../root/contact.md
- [HTML] /images/profile.png
============================================================
Total Images Found: 56
Total Files Containing Images: 17
============================================================
Change References to WebP
The six instances that were in the code I manually fixed those.
A modified version of the discovery script was used to replace the extensions in markdown.
import os
import re
from pathlib import Path
import argparse
# Common image extensions to be replaced
IMAGE_EXTENSIONS = r'(?:jpg|jpeg|png|gif|bmp|tiff)'
def replace_ext_to_webp(match):
"""
Callback function for re.sub.
It takes the match object and replaces the image extension with .webp
"""
full_match = match.group(0)
# Replace common extensions with .webp, case-insensitive
pattern = rf'\.(?:{IMAGE_EXTENSIONS})(?=[)\"\'\s])'
return re.sub(pattern, '.webp', full_match, flags=re.IGNORECASE)
def process_markdown_file(file_path, convert=False):
"""
Scans for images. If convert=True, replaces extensions and saves the file.
"""
# Regex for Markdown ![]() and HTML <img src="">
md_pattern = r'!\[.*?\]\((.*?)\)'
html_pattern = r'<img[^>]+src=["\']([^"\']+)["\'][^>]*>'
images_found = []
modified = False
try:
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
# 1. Identification (for summary purposes)
md_matches = re.findall(md_pattern, content)
for match in md_matches:
images_found.append({'type': 'Markdown', 'url': match})
html_matches = re.findall(html_pattern, content)
for match in html_matches:
images_found.append({'type': 'HTML', 'url': match})
# 2. Conversion (if requested)
if convert:
# We use re.sub with a callback to ensure we only change extensions
# inside the image tags, not in regular links or text.
new_content = re.sub(md_pattern, replace_ext_to_webp, content)
new_content = re.sub(html_pattern, replace_ext_to_webp, new_content)
if new_content != content:
with open(file_path, 'w', encoding='utf-8') as f_out:
f_out.write(new_content)
modified = True
except Exception as e:
print(f"Error processing {file_path}: {e}")
return images_found, modified
def scan_directory(root_dir, convert=False):
"""
Recursively searches for .md files and processes them.
"""
root_path = Path(root_dir)
if not root_path.is_dir():
print(f"Error: {root_dir} is not a valid directory.")
return None, 0, 0
summary = {}
total_images = 0
files_modified = 0
for md_file in root_path.rglob('*.md'):
images, was_modified = process_markdown_file(md_file, convert)
if images:
summary[str(md_file)] = images
total_images += len(images)
if was_modified:
files_modified += 1
return summary, total_images, files_modified
def print_summary(summary, total_count, modified_count, convert_mode):
"""
Formats and prints the final discovery and action report.
"""
print("\n" + "="*60)
print(f"IMAGE DISCOVERY {'& CONVERSION' if convert_mode else 'SUMMARY'}")
print("="*60)
if not summary:
print("No images found in any markdown files.")
return
for file, images in summary.items():
print(f"\nš File: {file}")
for img in images:
print(f" - [{img['type']}] {img['url']}")
print("\n" + "="*60)
print(f"Total Images Found: {total_count}")
print(f"Total Files Containing Images: {len(summary)}")
if convert_mode:
print(f"Total Files Updated to .webp: {modified_count}")
print("="*60 + "\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Identify and optionally convert image extensions in Markdown files.")
parser.add_argument("directory", nargs="?", default=".", help="The directory to scan (defaults to current directory)")
parser.add_argument("--convert-to-webp", action="store_true", help="Change all discovered image extensions to .webp")
args = parser.parse_args()
mode_text = " (Conversion Mode)" if args.convert_to_webp else " (Read-only Mode)"
print(f"Scanning directory: {os.path.abspath(args.directory)}...{mode_text}")
results, count, modified = scan_directory(args.directory, args.convert_to_webp)
if results is not None:
print_summary(results, count, modified, args.convert_to_webp)
I simply ran the same script, to the same location but added the "--convert-to-webp" parameter. It seems like the resulting math is off, but, it appeared to process correctly.
There is one additional factor to consider which each post. There are references to image cards in the header details of each markdown. I am going to manually edit those, since some need to be changed (more of a legacy issue during the ghost migration). After that, we test.
Test and Verify
Make sure when you are doing changes like this use version control and create a new branch. If you goof, you are in trouble. I pushed this to my dev branch and deployed it into my kubernetes cluster.
Testing is fairly straight forward, since again there are three main streams of images, the posts, the watches and the UNESCO Tracker. The first test run is how I uncovered the missing "visits" as all the posts were broken images.
Sample and Statistics
Here are some of the fun statistics coming out of this exercise!
| Statistic | Value |
|---|---|
| Total Original Size | 77.76 MB |
| Total WebP Size | 13.21 MB |
| Total Saved | 64.55 MB |
| Percentage Reduced | 83% |
| Largest Reduction By % | 96.5% |
| Worst Reduction By % | -12.8% |
| Largest Savings | 4.8 MB |
| Worst Savings | -0.32 KB |
The original size of the build container was 123 MBs. The new size after converting the images is 59 MBs. That is a crazy jump. It is way more than I thought. I am hoping to minimize the site and container more by doing additional enhancements to the container itself. Ideally, I want to have a "green" version of the site that can run on renewable energy; something like an ESP32 or Pi Pico. That's for another post.
