Ldiff Guide

ldiff is a line difference visualizer that processes input lines, replacing repeated tokens with a substitute character for easy pattern recognition. It’s particularly useful for analyzing log files, command output, and any text with recurring patterns.

Core Concepts

Pattern Recognition

ldiff reads input line by line and compares each line to the previous one. When it finds tokens (words, numbers) that repeat from the previous line, it replaces them with a substitute character, making unique content stand out.

Token Types

The tool recognizes several types of tokens:

• Words: Alphanumeric sequences (hello, world, API)
• Numbers: Numeric sequences (2023, 404, 127)
• Separators: Punctuation and symbols (:, -, /, @, etc.)
• Whitespace: Spaces, tabs, and other whitespace
• ANSI Codes: Color codes and terminal escape sequences

Preservation

While replacing repeated tokens, ldiff carefully preserves:

• ANSI color codes - Terminal colors remain intact
• Separators and punctuation - Structure is maintained
• Whitespace - Original spacing and alignment
• Case sensitivity - Hello and hello are different tokens

Basic Operations

Simple Usage

# Read from stdin with default substitute character (░)
echo -e "hello world\nhello universe" | wsb ldiff
# Output:
# hello world
# ░░░░░ universe

Custom Substitute Character

# Use a custom character for substitution
echo -e "test line\ntest another" | wsb ldiff "*"
# Output:
# test line
# **** another

Reading from Files

# Process a file
wsb ldiff < input.txt

# Process compressed logs
gunzip -c app.log.gz | wsb ldiff

# Use with other commands
cat /var/log/system.log | tail -n 100 | wsb ldiff

Log Analysis

System Logs

# Analyze system logs for patterns
tail -f /var/log/syslog | wsb ldiff

# Find repeated patterns in auth logs
grep "Failed password" /var/log/auth.log | wsb ldiff

Application Logs

# Monitor application logs in real-time
tail -f /var/log/myapp.log | wsb ldiff

# Analyze historical logs
cat /var/log/myapp.log.1 | wsb ldiff > patterns.txt

Web Server Logs

# Analyze access patterns
cat /var/log/nginx/access.log | wsb ldiff "■"

# Find common request patterns
awk '{print $7}' /var/log/apache2/access.log | sort | wsb ldiff

Example Log Analysis

# Input log lines:
# 2023-01-01 10:00:00 INFO Starting application
# 2023-01-01 10:00:01 INFO Loading configuration
# 2023-01-01 10:00:02 ERROR Failed to connect

cat app.log | wsb ldiff
# Output:
# 2023-01-01 10:00:00 INFO Starting application
# ░░░░-░░-░░ ░░:░░:01 ░░░░ Loading configuration
# ░░░░-░░-░░ ░░:░░:02 ERROR Failed to connect

Command Analysis

File Listings

# Analyze directory structures
find /usr/local -type f | wsb ldiff

# Compare file paths
ls -la /usr/bin/ | wsb ldiff "●"

Process Lists

# Analyze running processes
ps aux | wsb ldiff

# Monitor process changes
watch -n 1 "ps aux | head -20" | wsb ldiff

Network Analysis

# Analyze network connections
netstat -tulpn | wsb ldiff

# Monitor network activity
ss -tulpn | wsb ldiff "█"

Advanced Features

Unicode Support

# Use Unicode characters for substitution
echo -e "hello world\nhello universe" | wsb ldiff "█"
echo -e "test line\ntest another" | wsb ldiff "▓"
echo -e "data point\ndata value" | wsb ldiff "◆"

Piping and Redirection

# Save patterns to file
cat large.log | wsb ldiff > patterns.txt

# Combine with other tools
cat access.log | grep "GET" | wsb ldiff | head -50

# Chain multiple filters
journalctl -f | grep "ERROR" | wsb ldiff "*" | tee error_patterns.log

Real-time Monitoring

# Monitor multiple log sources
tail -f /var/log/app1.log /var/log/app2.log | wsb ldiff

# Monitor system activity
dmesg -w | wsb ldiff

# Watch command output
watch -n 2 "df -h" | wsb ldiff

Workflow Examples

Development Debugging

# 1. Start monitoring your application logs
tail -f /var/log/myapp.log | wsb ldiff &

# 2. Reproduce the issue
./trigger_bug.sh

# 3. Analyze the pattern differences
# Repeated timestamp/process info is masked, errors stand out

# 4. Save interesting patterns
pkill tail
cat /var/log/myapp.log | tail -100 | wsb ldiff > debug_patterns.txt

System Administration

# 1. Monitor system health
tail -f /var/log/syslog | wsb ldiff "■" &

# 2. Check for unusual patterns
# Normal startup messages will show patterns
# Anomalies will show unique content

# 3. Investigate specific issues
grep "$(date +%Y-%m-%d)" /var/log/syslog | wsb ldiff

Performance Analysis

# 1. Capture performance data
top -b -n 10 | wsb ldiff > perf_patterns.txt

# 2. Analyze patterns
cat perf_patterns.txt | grep -v "░"

# 3. Find anomalies
# Look for lines without substitute characters

Configuration Patterns

Log Timestamp Formats

ldiff automatically handles various timestamp formats:

# ISO 8601: 2023-01-01T10:00:00Z
# Syslog: Jan 01 10:00:00
# Apache: [01/Jan/2023:10:00:00 +0000]
# Custom: 2023-01-01 10:00:00.123

Common Separators

The tool recognizes these separator patterns:

• Paths: /usr/local/bin → /░░░/░░░░░/bin
• URLs: https://api.example.com → ░░░░░://░░░.░░░░░░░.com
• Email: user@domain.com → ░░░░@░░░░░░.com
• IPs: 192.168.1.100 → ░░░.░░░.░.░░░

Integration with Other Tools

With grep

# Filter then analyze patterns
grep "ERROR" /var/log/app.log | wsb ldiff

With awk

# Extract specific fields then analyze
awk '{print $1, $7}' /var/log/access.log | wsb ldiff

With sort/uniq

# Find unique patterns
cat /var/log/app.log | wsb ldiff | sort | uniq -c

With head/tail

# Analyze recent entries
tail -100 /var/log/app.log | wsb ldiff

# Check first patterns
head -50 /var/log/app.log | wsb ldiff

Safety Features

Signal Handling

• Broken Pipe: Gracefully handles when output is piped to head, less, etc.
• Keyboard Interrupt: Clean exit on Ctrl+C
• Input Validation: Validates substitute character input

Error Handling

• No Input: Clear error message when no stdin is provided
• Invalid Character: Warnings for multi-character substitutes
• File Permissions: Appropriate error messages for access issues

Best Practices

Performance

1. Large Files: Use with tail or head for very large files
2. Real-time: Use -f flag with tail for live monitoring
3. Memory: Tool processes line-by-line, memory usage is minimal

Analysis

1. Start Simple: Begin with default substitute character
2. Use Contrast: Choose substitute characters that stand out
3. Save Patterns: Redirect output to files for later analysis
4. Combine Tools: Use with grep, awk, and other text tools

Troubleshooting

1. No Patterns: If no substitution occurs, lines may be completely different
2. Too Much Substitution: Try processing smaller chunks
3. ANSI Issues: Some terminals may not display substitute characters correctly

Common Use Cases

Security Monitoring

# Monitor failed login attempts
tail -f /var/log/auth.log | grep "Failed" | wsb ldiff

# Analyze access patterns
cat /var/log/nginx/access.log | wsb ldiff | grep -v "░"

Application Monitoring

# Track API response patterns
tail -f /var/log/api.log | grep "response_time" | wsb ldiff

# Monitor database queries
tail -f /var/log/mysql/slow.log | wsb ldiff

System Analysis

# Analyze process startup patterns
dmesg | grep "systemd" | wsb ldiff

# Monitor resource usage patterns
vmstat 1 | wsb ldiff

See Also

• Installation Guide - How to install ldiff
• Usage Guide - General usage examples
• Scrap Guide - Manage temporary files
• Examples - More real-world examples