Lightweight ASCII English Text Stream Compression in Python.

NOTE : updated documentation and source code is available at :

https://github.com/rodv92/PLETSC

PLETSC

Lightweight english text stream compression, with word tokens, ngrams, session dictionaries and Huffmann for unknown words.

How to use :

git clone and decompress dics.zip in the current folder.

Syntax for compression :

python3 dicstrv.py -c txt_inputfile compressed_outputfile Reads txt_inputfile and writes compressed text stream to compressed_outputfile.

python3 dicstrv.py -c txt_inputfile Reads txt_input file and writes compressed output to stdout

Syntax for decompression :

python3 dicstrv.py -x compressed_inputfile txt_outputfile Reads compressed_inputfile and writes cleartext to txt_outputfile.

python3 dicstrv.py -x compressed_inputfile

Reads compressed_input file and writes cleartext output to stdout

Syntax to generate a compiled dictionary of ngrams :

python3 dicstrv.py -d cleartext_ngrams_inputfile compressed_ngrams

This is rarely used in normal operation.

NOTE: dictionary file count1_w.txt must be in the same directory as the script. outngrams.bin must be in the same directory as the script, if ngrams are used (secondpass=True)

Description :

This script is useful for ASCII English text stream compression. It’s pedantic (P in PLETSC stands for “pedantic”) because its final goal is to enforce a minima some English syntactic rules, such as whitespace after “,” but not before, Capitalization after a “.” etc… (but not grammar). Spell check will probably be recommended but should probably be done upstream (by another applicative layer), as it will ensure a better compression ratio – since it is based on words of the english dictionary.

Its compression method is primarily based on a token (words and punctuation) dictionary. It leverages frequency of modern english words:

  • Words of the primary dictionary are sorted from most used to least used.
  • The line number is used as an index. (+1) index 0 is reserved for whitespace.

It also uses adaptive length encoding (1-2-3 bytes) First 128 most used tokens are encoded on 1 byte, Next 16384 + 128 on 2 bytes. Next 2097152 + 16384 + 128 on 3 bytes.

The 3 byte address space is split in two :

  • First part (when byte0 msb is 1 and byte1 msb is 1 and byte2 msb is 0) is further divided into two subspaces.
    • The first subspace is for the remainder of the primary dictionary (it has 333 333 tokens).
    • And the second subspace holds an Ngram dictionary (more on that later).
  • Second part (when byte0 msb is 1 and byte1 msb is 1 and byte2 msb is 1) is further divided into two subspaces.
    • First part is for a session dictionary. A session dictionary is used to hold repeating unknown tokens. there are 2097152 – 5 codes available for this use. Initially empty. Kept in ram, it is a SESSION dictionary. This session dictionary should not be required to be sent between two parties, as it can be reconstructed entirely from the compressed stream.
    • Second part is only 5 codes, (TODO, for now just 1 code, and switch between Huffmann and no compression is done in a bool parameter) It is an escape sequence meaning that following bytes will be encoded wit the following methods :
      • first code : As a stream of chars (no compression), plus a C style termination (chr(0)).
      • second code : Huffmann encoding, lowercase only.
      • third code : Huffmann, lowercase + uppercase or uppercase only.
      • fourth code : Huffmann, lowercase + uppercase + numbers, or numbers only.
      • fifth code : All printable ASCII space, mainly for passwords. Each of these codes tells what Huffmann tree to use.

Performance :

It offers a good compression ratio (between 2.6 and 3.0+), That is, Sizes in % of ORIGINAL size of around 33% to 38%, mainly depending on the lexical complexity or lexical archaism of the source text, and presence of unkwnown or misspelled words.

A higher lexical complexity, or archaic texts, that is, if the input text uses less common words – based on current usage – (2023), will yield lower compression ratios.

The compresion ratio is more or less stable : it is quite independent of text length.

This is contrary to block algorithms that suffer from low compression for small files because of a significant overhead. For larger corpuses, block algorithms usually perform better, and modern methods may use ML methods to provide context and adaptive encoding based on that, they’re usually slower.

This is why this algorithm is intended for stream compression (on the fly). However, its current implementation is based on reading files. and outputting to a file or stdout.

Compression speed (all options enabled)

For this test :

  • File is call_of_cthulhu.txt, size uncompressed is 69 kB
  • Compression speed around 23,3 kB/s on a Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz (computer from 2011), + SSD storage

Footprint (filesystem)

zipped size of count1_w.txt + outngrams.bin is 11 566 806 bytes unzipped size is : 31 327 633 bytes + 3 157 445 bytes = 34 485 078 bytes.

Footprint (memory)

To be determined

Dependecies

These Python modules are required :

codecs, nltk, re, bitstring, bitarray, struct, time, dahuffman

Requirements

Input text file must be ASCII (for now) or UTF-8 decodable to ASCII (English). It ignores conversion errors. Decoded file will be encoded in ASCII. It should be in English to get adequate conversion.

Both ends (sender and receiver) MUST have the SAME dictionaries and the SAME Huffmann tables, as these are not sent with the data.

Information about the dictionaries

The primary dictionary is based on the “count_1w.txt” english dictionary of 333 333 words, (words ordered by lexical prevalence) tweaked with added special characters also listed by order of prevalence and added english contractions, and with word count number stripped off.

The original primary dictionary file is available on : https://norvig.com/ngrams/

It also features a secondary (optional) compression pass based on a compiled dictionary named outngrams.bin.

It features compression for 4 and 5 word ngrams found in the first compression step stream. Ngrams of less than 4 words are deemed not interesting as the first pass will usually encode them on 3 bytes, the same sized as a compressed ngram.

Compression and decompression require the primary dictionary to be available, and the secondary if the boolean SecondPass is set to true, (by default).

The zip “dics.zip” already have a compiled version of these dictionaries.

More information

The algorithm is heavily commented in the code.

Field of application

Main applications could be messaging over low bandwidth links like POCSAG radio text, or JS8Call for HAM radio, and IoT.

However, note that the underlying digital mode should allow binary transmission (not only transmission of printable ASCII characters) for seamless integration.

TODO and ISSUES :

See comments in the code.

Main issues for now are syntactic rules and spurious whitespaces, absence of whitespaces were they should have been, problems with hyphenated tokens, spurious newlines, problems with some possessive forms, and special constructs besides emails and well formed URLs.

Ngrams Processing from scratch :

Useful if you want to tweak or create your own dictionaries, we’ll discuss mainly the outngrams.bin dictionary, as count_1w.txt tweaking is straightforward. Note that count1_w.txt should not be modified once outngrams.bin is generated, or you’ll have to rebuild outngrams.bin

A preparatory step is required to generate a compressed version of the ngrams files, if you want to do it from scratch :

First create the ngrams CSV using this code repo : https://github.com/orgtre/google-books-ngram-frequency/tree/main/python

The repo contains scripts that perform the download and concatenation of ngrams according to criterions you specify. Note that LETSC has limited space in the first subspace of the 3 byte. more or less 2097152 – 333333 I have created an ngram list of 1571125 ngrams. The distribution between the 4grams and 5grams is roughly 50%/50%

The resulting CSV files need to be further processed by our algorithm

The script that create outngrams.bin (the secondary compiled dictionary based on the primary dictionary and the ngrams csv files from google-books-ngram) is called ngrams_format_dic.py This script is commented for what each line does.

# LIGHTWEIGHT ENGLISH TEXT STREAM COMPRESSION (LETSC)
# (adaptive encoding length 1byte/2byte/3byte based on word dictionary with statistical prevalence ordering - count1_w.txt)
# Huffmann encoding for uknown tokens
# Enforces English syntax rules for punctuation
# Takes into account possessives and contractions
# Has URLs and e-mails processing rules, more to follow
# Second pass compression using a dictionary of the most frequent 4 N-Grams of English fiction.

#GPL 3 License
# www.skynext.tech
# Rodrigo Verissimo
# v0.92
# October 21th, 2023


# Python + packages Requirements

# Python 3.9
# nltk, bitarray, bitstring, re, dahuffmann

# Performance : ratios between x2.6 for Middle to Modern and elaborate English (ex: Shakespeare)
# Up to x3 and more for simple english.
# adapted for text messaging / streaming
# Requires the same dictionary on both channel endpoints.

# ALGORITHM. Very straightforward. (adaptive encoding length based on dictionary with statistical ordering)

#################################################################################
# First byte :

#if MSB is 0, a whole word is encoded on the first 7 bits of one byte only.
#This makes 127 possible words. These are reserved for the first 127 most used 
#english words. punctuation also appears as a possible word

# Second byte :

#if MSB of first byte is 1, and MSB of second byte is 0, a whole word is encoded
# with the help of the 7 LSB of byte 1 plus the 7 LSB of byte 2. 
# This makes room for the next 16384 most used english words.

# Third byte :
# if MSB of first byte is 1 and MSB of second byte is 1, and the MSB of third byte is 0
# a whole word is encoded
# with the help of the 7 + 7 + 7 = 21 bits (2 097 152 possible words)

# For now, the 3 byte address space is split into two 2 097 152 address spaces
# That is, the case of all 3 bytes MSB being 1 is treated separately.
# In this adress space, only a handful of codes are used as an escape sequence for particular 
# Huffmann trees, see below.

#->
#load dictionary of english words from most used to least used.
#punctuation and special characters have been added with order of prevalence.
#punctuation frequency is from wikipedia corpus. (around 1.3 billion words) 
#it has been normalized to the frequency of the 1/3 million word list based 
#on the google web trillon word corpus. that is, frequencies for special chars have been multiplied by 788.39
#wikipedia punctuation is not optimized for chat, as it lower prevalence of chars like question marks
#that may appear more frequently in chat situations.

# the first tokenizer used does not separate any special character attached (without whitespace) to a word
# this will mostly result in an unknown word in the dictionary
# this key absence in the reverse dict will be catched and treated by another tokenizer (mainly for possessive
# forms and contractions)

#for possessives ex: "dog's bone" or plural "dogs' bones" a separate tokenizer is used to split into
# "dog" , "'s"
# "'s" and "'" also appear in the dictionary.

# ROADMAP
# remove whitespaces left of punctuation DONE
# manage new lines DONE
# manage websites and emails DONE
# TODO
# add spell check ! 
# TODO
# Remove spurious new lines that appear after encoding special sequences such mails or URLS
# DONE (basic Huffmann, some chars missing in tree)
# add Huffmann encoding for absent words in dictionary (neologisms,colloqualisms,dialects, or misspellings) DONE
# DONE

# TODO : test with more texts such as wikipedia XML and various authors works, to catch as much
# use cases and formatting issues that arise to improve the algorithm

# add adaptive Huffmann. use 4 Huffmann trees. (see below)
# Assuming there are 4 codes for hufmmann : hufmann lower case, hufmann lower + capitals, huffmann
# lower + capitals + numeric, all printable ASCII excluding whitespace : same as preceding category plus 
# special chars.
# Chosing the tree to use would be done by string regex.

#DONE
# Detect UTF-8 and transcode to ASCII (potentially lossy)
#DONE


# TODO
# Dictionary Learn over time (re-shuffle the order of tokens)
# Without transmission of any info between parties
# Dangerous if sync is lost between the two parties
# TODO

# TODO
# optimize Huffmann part to remove the need for the chr(0) termination = scan for EOF sequence in Huffmann to get
# the Huffmann byte sequence length. TODO


# DONE
# Add second pass compression using word N-grams lookup table. (4 and 5 N-grams seem to be a good compromize)
# The idea is to encode 4 and 5 token substrings in a line by a single 3 byte code.
# There is plenty of room left in the 3 byte address space. For now, there is 333 333 - 16384 - 128 tokens used = 316821 tokens used
# from 4194304 - 3 total address space.
# DONE using 1 571 125 codes for a 50/50 mix of 4grams and 5grams.
# There is still at least 2million codes left.
#  for now we plan 4 escape sequences for the selection of one of the 4 Huffmann trees.


# ngrams processing is first done with the create_ngrams_dic.sh script.
"""
python3 ngrams_format_dic.py 4grams_english-fiction.csv outngrams4.txt #remove counts and process contractions
python3 ngrams_format_dic.py 5grams_english-fiction.csv outngrams5.txt #remove counts and process contractions

python3 dicstrv4.py -d outngrams4.txt outngrams4.bin.dup #convert ngrams txt to compressed form
python3 dicstrv4.py -d outngrams5.txt outngrams5.bin.dup #convert ngrams txt to compressed form
awk '!seen[$0]++' outngrams4.bin.dup > outngrams4.bin #Remove spurious duplicates that may arise
awk '!seen[$0]++' outngrams5.bin.dup > outngrams5.bin #Remove spurious duplicates that may arise
sed -i '786001,$ d' outngrams4.bin # truncate to fit target address space
sed -i '786001,$ d' outngrams5.bin # truncate to fit target address space

cat outngrams4.bin outngrams5.bin > outngrams.bin # concatenate. this is our final form
cat outngrams.bin | awk '{ print length, bash $0 }' | sort -n -s | cut -d" " -f2- > sorted.txt # sort by size to have an idea of distribution

# ngrams that encode as less than 4 bytes have been pruned since the ratio is 1

"""

# DONE 
# It is probable that the most used 4 tokens N-grams are based on already frequent words. that individually
# encode as 1 byte or two bytes.
# Worst case : all the 4 tokens are encoded in the 1 to 128 addres space, so they take a total 4 bytes.
# The resulting code will be 3 bytes, a deflate percent of 25%
# If one of the tokens is 2 byte (128 to 16384 -1 address space), then it uses 5 bytes.
# deflate percent is 40%
# The unknown is the statistical prevalence of two million 4 token N-grams.
# (ex: coming from english fiction corpus) in a standard chat text.

# First encode the google most frequent 4 and 5 N-grams csv file to replace the tokens in each N-gram by the corrsponding 
# byte sequences from our codes in the count_1w.txt dictionary. This will be another pre-process script.
# The resulting new csv format will be :
# some 3 byte index = x04x09x23.
# The 3 byte index is simply the line number of the compressed ngram. 

# read that in ram. Conservative Estimate 4 bytes + 3 bytes per entry 7 bytes * 2 000 000 = 14 Meg memory footprint.
# We already have a 4 MB * 3  12 Meg footprint from count_1w (estimate)

# Generate the inverse map dictionary (mapping sequences to 3 byte indexes)
# x04x09x23' = some 3 byte index
# Should not be a problem since there is a 1 to 1 relationship between the two

# Then perform a first pass compression.
# Then scan the first pass compression file using a 4 token sliding window.
# Contractions is a case that will have to be managed.

# If there are overlapping matches, chose the match that result in the best deflation, if any.
# If the unknown word escape codes appears, stop processing and resume after the escaped word

# Overall, replace the byte sequence by the corrsponding 3 byte sequence.
# DONE



import sys
import traceback

#print(len(sys.argv))
#op = (sys.argv[1]).encode("ascii").decode("ascii")
#print(op)
#quit()

if ((len(sys.argv) < 3) or (len(sys.argv) > 4)):
    print("Syntax for compression :\n")
    print("python3 dicstrv.py -c <txt_inputfile> <compressed_outputfile>")
    print("Reads txt_inputfile and writes compressed text stream to compressed_outputfile.\n") 
    
    print("python3 dicstrv.py -c <txt_inputfile>")
    print("Reads txt_input file and writes compressed output to stdout\n")

    print("Syntax for decompression :\n")
    print("python3 dicstrv.py -x <compressed_inputfile> <txt_outputfile>")
    print("Reads compressed_inputfile and writes cleartext to txt_outputfile.\n") 
    
    print("python3 dicstrv.py -x <compressed_inputfile>\n")
    print("Reads compressed_input file and writes cleartext output to stdout\n")

    print("NOTE: dictionary file count1_w.txt must be in the same directory as the script.")    
    quit()

if (sys.argv[1] == "-c"):
    compress = True
    gendic = False
elif (sys.argv[1] == "-d"):
    compress = True
    gendic = True
elif (sys.argv[1] == "-x"):
    compress = False
    gendic = False
else:
    print("unknown operation: " + str(sys.argv[0]) + " type 'python3 dicstrv3.py' for help")

if (len(sys.argv) == 3):
    infile = sys.argv[2]
    outfile = ''
if (len(sys.argv) == 4):
    infile = sys.argv[2]
    outfile = sys.argv[3]

import codecs
import nltk
from nltk.tokenize import TweetTokenizer
tknzr = TweetTokenizer()

import re
import bitstring
from bitarray import bitarray
import struct
import time
from dahuffman import HuffmanCodec


debug_on = False
debug_ngrams_dic = False
secondpass = True
use_huffmann = False
unknown_token_idx = 16384 + 128 + 2097152


def debugw(strdebug):
    if (debug_on):
        print(strdebug)

# Huffmann is only used for absent words in count1_w.txt dictionary
# General lower and upper case frequency combined as lowercase



codec_lower = HuffmanCodec.from_frequencies(
{'e' :   56.88,	'm' :	15.36,
'a'	:	43.31,	'h'	:	15.31,
'r'	:	38.64,	'g'	:	12.59,
'i'	:	38.45,	'b'	:	10.56,
'o'	:	36.51,	'f'	:	9.24,
't'	:	35.43,	'y'	:	9.06,
'n'	:	33.92,	'w'	:	6.57,
's'	:	29.23,	'k'	:	5.61,
'l'	:	27.98,	'v'	:	5.13,
'c'	:	23.13,	'x'	:	1.48,
'u'	:	18.51,	'z'	:	1.39,
'd'	:	17.25,	'j'	:	1,
'p'	:	16.14,	'q'	:	1
}
)

debugw(codec_lower.get_code_table())

# following is ASCII mixed upper and lower case frequency from an English writer from Palm OS PDA memos in 2002
# Credit : http://fitaly.com/board/domper3/posts/136.html

codec_upperlower = HuffmanCodec.from_frequencies(

{'A' : 0.3132,
'B' : 0.2163,
'C' : 0.3906,
'D' : 0.3151,
'E' : 0.2673,
'F' : 0.1416,
'G' : 0.1876,
'H' : 0.2321,
'I' : 0.3211,
'J' : 0.1726,
'K' : 0.0687,
'L' : 0.1884,
'M' : 0.3529,
'N' : 0.2085,
'O' : 0.1842,
'P' : 0.2614,
'Q' : 0.0316,
'R' : 0.2519,
'S' : 0.4003,
'T' : 0.3322,
'U' : 0.0814,
'V' : 0.0892,
'W' : 0.2527,
'X' : 0.0343,
'Y' : 0.0304,
'Z' : 0.0076,
'a' : 5.1880,
'b' : 1.0195,
'c' : 2.1129,
'd' : 2.5071,
'e' : 8.5771,
'f' : 1.3725,
'g' : 1.5597,
'h' : 2.7444,
'i' : 4.9019,
'j' : 0.0867,
'k' : 0.6753,
'l' : 3.1750,
'm' : 1.6437,
'n' : 4.9701,
'o' : 5.7701,
'p' : 1.5482,
'q' : 0.0747,
'r' : 4.2586,
's' : 4.3686,
't' : 6.3700,
'u' : 2.0999,
'v' : 0.8462,
'w' : 1.3034,
'x' : 0.1950,
'y' : 1.1330,
'z' : 0.0596
})

debugw(codec_upperlower.get_code_table())

# following is ASCII alpha numeric frequency from an English writer from Palm OS PDA memos in 2002
# Credit : http://fitaly.com/board/domper3/posts/136.html

codec_alphanumeric = HuffmanCodec.from_frequencies(

{'0' : 0.5516,
'1' : 0.4594,
'2' : 0.3322,
'3' : 0.1847,
'4' : 0.1348,
'5' : 0.1663,
'6' : 0.1153,
'7' : 0.1030,
'8' : 0.1054,
'9' : 0.1024,
'A' : 0.3132,
'B' : 0.2163,
'C' : 0.3906,
'D' : 0.3151,
'E' : 0.2673,
'F' : 0.1416,
'G' : 0.1876,
'H' : 0.2321,
'I' : 0.3211,
'J' : 0.1726,
'K' : 0.0687,
'L' : 0.1884,
'M' : 0.3529,
'N' : 0.2085,
'O' : 0.1842,
'P' : 0.2614,
'Q' : 0.0316,
'R' : 0.2519,
'S' : 0.4003,
'T' : 0.3322,
'U' : 0.0814,
'V' : 0.0892,
'W' : 0.2527,
'X' : 0.0343,
'Y' : 0.0304,
'Z' : 0.0076,
'a' : 5.1880,
'b' : 1.0195,
'c' : 2.1129,
'd' : 2.5071,
'e' : 8.5771,
'f' : 1.3725,
'g' : 1.5597,
'h' : 2.7444,
'i' : 4.9019,
'j' : 0.0867,
'k' : 0.6753,
'l' : 3.1750,
'm' : 1.6437,
'n' : 4.9701,
'o' : 5.7701,
'p' : 1.5482,
'q' : 0.0747,
'r' : 4.2586,
's' : 4.3686,
't' : 6.3700,
'u' : 2.0999,
'v' : 0.8462,
'w' : 1.3034,
'x' : 0.1950,
'y' : 1.1330,
'z' : 0.0596
})

debugw(codec_alphanumeric.get_code_table())

# following is Whole ASCII printable chars frequency except whitespace from an English writer from Palm OS PDA memos in 2002
# Credit : http://fitaly.com/board/domper3/posts/136.html

codec_all = HuffmanCodec.from_frequencies(

{'!' : 0.0072,
'\"' : 0.2442,
'#' : 0.0179,
'$' : 0.0561,
'%' : 0.0160,
'&' : 0.0226,
'\'' : 0.2447,
'(' : 0.2178,
')' : 0.2233,
'*' : 0.0628,
'+' : 0.0215,
',' : 0.7384,
'-' : 1.3734,
'.' : 1.5124,
'/' : 0.1549,
'0' : 0.5516,
'1' : 0.4594,
'2' : 0.3322,
'3' : 0.1847,
'4' : 0.1348,
'5' : 0.1663,
'6' : 0.1153,
'7' : 0.1030,
'8' : 0.1054,
'9' : 0.1024,
':' : 0.4354,
';' : 0.1214,
'<' : 0.1225,
'=' : 0.0227,
'>' : 0.1242,
'?' : 0.1474,
'@' : 0.0073,
'A' : 0.3132,
'B' : 0.2163,
'C' : 0.3906,
'D' : 0.3151,
'E' : 0.2673,
'F' : 0.1416,
'G' : 0.1876,
'H' : 0.2321,
'I' : 0.3211,
'J' : 0.1726,
'K' : 0.0687,
'L' : 0.1884,
'M' : 0.3529,
'N' : 0.2085,
'O' : 0.1842,
'P' : 0.2614,
'Q' : 0.0316,
'R' : 0.2519,
'S' : 0.4003,
'T' : 0.3322,
'U' : 0.0814,
'V' : 0.0892,
'W' : 0.2527,
'X' : 0.0343,
'Y' : 0.0304,
'Z' : 0.0076,
'[' : 0.0086,
'\\' : 0.0016,
']' : 0.0088,
'^' : 0.0003,
'_' : 0.1159,
'`' : 0.0009,
'a' : 5.1880,
'b' : 1.0195,
'c' : 2.1129,
'd' : 2.5071,
'e' : 8.5771,
'f' : 1.3725,
'g' : 1.5597,
'h' : 2.7444,
'i' : 4.9019,
'j' : 0.0867,
'k' : 0.6753,
'l' : 3.1750,
'm' : 1.6437,
'n' : 4.9701,
'o' : 5.7701,
'p' : 1.5482,
'q' : 0.0747,
'r' : 4.2586,
's' : 4.3686,
't' : 6.3700,
'u' : 2.0999,
'v' : 0.8462,
'w' : 1.3034,
'x' : 0.1950,
'y' : 1.1330,
'z' : 0.0596,
'{' : 0.0026,
'|' : 0.0007,
'}' : 0.0026,
'~' : 0.0003,
})

debugw(codec_all.get_code_table())
#quit()        

def check_file_is_utf8(filename):
    debugw("checking encoding of:")
    debugw(filename)
    try:
        f = codecs.open(filename, encoding='utf-8', errors='strict')
        for line in f:
            pass
        debugw("Valid utf-8")
        return True
    except UnicodeDecodeError:
        debugw("invalid utf-8")
        return False

def find_huffmann_to_use(token):

    if(not use_huffmann):
        debugw("do not use Huffmann, encode char by char")
        return 0
    
    not_alllower = re.search("[^a-z]")
    
    if(not not_alllower):
        debugw("all lower case")
        return 1
    
    not_alllowerorupper = re.search("[^A-Za-z]")
    
    if(not not_alllowerorupper):
        debugw("all lower or upper")
        return 2
    
    not_alllalphanumeric = re.search("[^A-Za-z0-9]")
    
    if(not not_alllalphanumeric):
        debugw("all alpha numeric")
        return 3
    else:
        debugw("all printable, except whitespace")
        return 4
    
def encode_unknown(token,treecode):

    if (treecode == 0):
        bytes_unknown = bytearray()
        for charidx in range(0, len(token)):
            debugw("appending chars..")
            debugw(token[charidx])

            # only append if it is not an unexpected termination in the unknown token
            if (not ord(token[charidx]) == 0):
                bytes_unknown.append(ord(token[charidx]))
            else:
                debugw("unexpected termination chr(0) in unknown token, discarding character")


        return bytes_unknown
    if (treecode == 1):
        return codec_lower.encode(token)
    if (treecode == 2):
        return codec_upperlower.encode(token)           
    if (treecode == 3):
        return codec_alphanumeric.encode(token)                      
    if (treecode == 4):
        return codec_all.encode(token)                      

def decode_unknown(bytetoken,treecode):

    if (treecode == 1):
        return codec_lower.decode(bytetoken)
    if (treecode == 2):
        return codec_upperlower.decode(bytetoken)           
    if (treecode == 3):
        return codec_alphanumeric.decode(bytetoken)                      
    if (treecode == 4):
        return codec_all.decode(bytetoken)  

def compress_token_or_subtoken(compressed,line_token,token_of_line_count,lentoken,gendic):
  
    
    global unknown_token_idx

    try:

        # is the token in english dictionary ?
        debugw("line_token:" + line_token)
        tokenid = engdictrev[line_token]
        subtokensid = [tokenid]

        
    except:
        debugw("unknown word, special chars adjunct, or possessive form")
        # let's try to split the unknown word from possible adjunct special chars
        # for this we use another tokenizer
        subtokens = nltk.word_tokenize(line_token)
        if (len(subtokens) == 1):
            # no luck...
            # TODO : do not drop the word silently, encode it !
            # If we encode a ngram dic, skip ngrams with unknown tokens in the primary dic.
            # and return empty bytearray to signify ngram compression failure 
            if(gendic):
                compressed = bytearray()
                debugw("gendic : unknown word")
                return (compressed, token_of_line_count)
        
            debugw("unknown word")

            #AMEND dictionary 
            # add this unknown subtoken to a session dic so it can be recalled.
            debugw("unknown word: " + subtokens[0] + " adding to session dic at id: " + str(unknown_token_idx))
            debugw("unknown word, adding to session dic at id: " + str(unknown_token_idx))
            
            engdictrev[subtokens[0]] = unknown_token_idx
            engdict[unknown_token_idx] = subtokens[0]
            unknown_token_idx += 1
                       

            #subtokensid = [4194304 - 1] # subtoken code for unknown word escape sequence.                       
            subtokensid = [4194303 - find_huffmann_to_use(subtokens[0])]                   
            #print(subtokensid)
            #continue
        else:
            debugw("possible special char found")
            subtokensid = []
            for subtoken in subtokens:
                debugw("subtoken=")
                debugw(subtoken)
                try:
                    subtokensid.append(engdictrev[subtoken])
                except:
                    # no luck...
                    # TODO : do not drop the word silently, encode it !
        
                    # If we encode a ngram dic, skip ngrams with unknown tokens in the primary dic.
                    # and return empty bytearray to signify ngram compression failure 
                    if(gendic):
                        compressed = bytearray()
                        debugw("gendic : unknown word")
                        return (compressed, token_of_line_count)
        
                    debugw("unknown subtoken")
                    subtokensid.append(4194303 - find_huffmann_to_use(subtoken))
                    #subtokensid.append(4194304 - 1)
                    
                    # add this unknown subtoken to a session dic so it can be recalled.
                    #AMEND dictionary 
                    # add this unknown subtoken to a session dic so it can be recalled.
                    debugw("unknown subtoken: " + subtoken + " adding to session dic at id: " + str(unknown_token_idx))
                    debugw("unknown subtoken, adding to session dic at id: " + str(unknown_token_idx))
                    engdictrev[subtoken] = unknown_token_idx
                    engdict[unknown_token_idx] = subtoken
                    unknown_token_idx += 1
                    #continue
    subtokenidx = 0
    for subtokenid in subtokensid:        
        
        debugw("subtokenid=")
        debugw(subtokenid)
        # maximum level of token unpacking is done
        if(subtokenid < 128):

            debugw("super common word")
            debugw(engdict[subtokenid])

            #convert to bytes
            byte0 = subtokenid.to_bytes(1, byteorder='little')
            debugw("hex:")
            debugw(byte0.hex())

            #append to bytearray
            compressed.append(byte0[0])

        if(128 <= subtokenid < 16384 + 128):

            debugw("common word")

            #remove offset
            debugw(engdict[subtokenid])
            subtokenid -= 128
            
            #convert to bytes1 (array of 2 bytes)
            bytes1 = subtokenid.to_bytes(2,byteorder='little')
            debugw("".join([f"\\x{byte:02x}" for byte in bytes1]))
        
            #convert to bitarray
            c = bitarray(endian='little')
            c.frombytes(bytes1)
            debugw(c)
            
            # set msb of first byte to 1 and shift the more significant bits up.
            c.insert(7,1)
            debugw(c)
            
            # remove excess bit
            del c[16:17:1]
            debugw(c)
            
            # append our two tweaked bytes to the compressed bytearray
            compressed.append((c.tobytes())[0])
            compressed.append((c.tobytes())[1])

        #if(16384 +128 <= subtokenid < 4194304 - 1):
        if(16384 +128 <= subtokenid < 2097152 + 16384 + 128):


            debugw("rare word")
            
            # remove offset
            debugw(engdict[subtokenid])
            subtokenid -= (16384 + 128)

            #convert to bytes1 (array of 3 bytes)
            bytes2 = subtokenid.to_bytes(3,byteorder='little')
            debugw("".join([f"\\x{byte:02x}" for byte in bytes2]))

            #convert to bitarray
            c = bitarray(endian='little')
            c.frombytes(bytes2)
            debugw(c)
            
            # set msb of first byte to 1 and shift the bits above up.
            c.insert(7,1)
            debugw(c)

            # set msb of second byte to 1 and shift the bits above up.
            c.insert(15,1)
            debugw(c)

            # remove two excess bits that arose from our shifts
            del c[24:26:1]
            debugw(c)
            
            # append our three tweaked bytes to the compressed bytearray
            compressed.append((c.tobytes())[0])
            compressed.append((c.tobytes())[1])
            compressed.append((c.tobytes())[2])


                #if(16384 +128 <= subtokenid < 4194304 - 1):
        if(16384 +128 + 2097152 <= subtokenid < 4194304 - 5):


            debugw("unknown word from session DIC")
            
            # remove offset
            debugw(engdict[subtokenid])
            subtokenid -= (2097152 + 16384 + 128)

            #convert to bytes1 (array of 3 bytes)
            bytes2 = subtokenid.to_bytes(3,byteorder='little')
            debugw("".join([f"\\x{byte:02x}" for byte in bytes2]))

            #convert to bitarray
            c = bitarray(endian='little')
            c.frombytes(bytes2)
            debugw(c)
            
            # set msb of first byte to 1 and shift the bits above up.
            c.insert(7,1)
            debugw(c)

            # set msb of second byte to 1 and shift the bits above up.
            c.insert(15,1)
            debugw(c)

            # set msb of third byte to 1 and shift the bits above up.
            c.insert(23,1)
            debugw(c)


            # remove three excess bits that arose from our shifts
            del c[24:27:1]
            debugw(c)
            
            # append our three tweaked bytes to the compressed bytearray
            compressed.append((c.tobytes())[0])
            compressed.append((c.tobytes())[1])
            compressed.append((c.tobytes())[2])


        #if(subtokenid == (4194304 - 1)):
        if(subtokenid in range(4194299,4194304)):

            #compressed.append(255)
            #compressed.append(255)
            #compressed.append(255)
            debugw("huffmann tree code :" + str(subtokenid))

            # TODO : Use Huffmann tree instead of byte->byte encoding.
            
            #convert to bytes1 (array of 3 bytes)
            bytes2 = subtokenid.to_bytes(3,byteorder='little')
            debugw("".join([f"\\x{byte:02x}" for byte in bytes2]))

            #convert to bitarray
            c = bitarray(endian='little')
            c.frombytes(bytes2)
            debugw(c)
            
            # set msb of first byte to 1 and shift the bits above up.
            c.insert(7,1)
            debugw(c)

            # set msb of second byte to 1 and shift the bits above up.
            c.insert(15,1)
            debugw(c)

            # no need to set  msb of third byte to 1 since the range will take care of it.
            #c.insert(23,1)
            #debugw(c)

            # remove two excess bits that arose from our shifts
            del c[24:26:1]
            debugw(c)
            
            # append our three tweaked bytes that signify the huffmann tree to use to the compressed bytearray
            compressed.append((c.tobytes())[0])
            compressed.append((c.tobytes())[1])
            compressed.append((c.tobytes())[2])

            if (len(subtokens) == 1):
                if(not use_huffmann):
                    debugw("encoding unkown word")
                    #for charidx in range(0, len(line_token)):
                    #    debugw("appending chars..")
                    #    debugw(line_token[charidx])
                    #    compressed.append(ord(line_token[charidx]))
                    compressed.extend(encode_unknown(line_token,0))
                else:
                    debugw("encoding unkown line token with Huffmann")
                    huffmann_tree_code = -(subtokenid - 4194303)
                    compressed.extend(encode_unknown(line_token,huffmann_tree_code))
            else:
                if(not use_huffmann):
                    debugw("encoding unkown subtoken")
                    #for charidx in range(0, len(subtokens[subtokenidx])):
                    #    debugw("appending chars..")
                    #    debugw((subtokens[subtokenidx])[charidx])
                    #    compressed.append(ord((subtokens[subtokenidx])[charidx]))
                    compressed.extend(encode_unknown(subtokens[subtokenidx],0))
                else:
                    debugw("encoding unkown subtoken with Huffmann")
                    debugw(subtokens[subtokenidx])
                    #huffmann_tree_code = find_huffmann_to_use(subtokens[subtokenidx])
                    huffmann_tree_code = -(subtokenid - 4194303)
                    compressed.extend(encode_unknown(subtokens[subtokenidx],huffmann_tree_code))
            compressed.append(0) # terminate c string style
        subtokenidx += 1        
    token_of_line_count += 1

    debugw("token of line count")
    debugw(token_of_line_count)
    debugw("lentoken")
    debugw(lentoken)

    if((token_of_line_count == lentoken) and (not gendic)):
        # newline
        debugw("append new line")
        compressed.append(0)
        #quit()  

    return (compressed,token_of_line_count)


def compress_tokens(tokens,gendic):

    #time.sleep(0.001)    
    # Init byte array
    compressed = bytearray()
    
    debugw("tokens are:")
    debugw(tokens)

    for token in tokens:

        debugw("token is:")
        debugw(token)

        token_of_line_count = 0
        # start compression run
        if(not len(token) and (not gendic)):
            debugw("paragraph")
            compressed.append(0)
            #compressed.append(0)
            #quit()
        lentoken = len(token)
        if (not gendic):
            for line_token in token:           
                (compressed, token_of_line_count) = compress_token_or_subtoken(compressed,line_token,token_of_line_count,lentoken,gendic)
        else:
                (compressed, token_of_line_count) = compress_token_or_subtoken(compressed,token,token_of_line_count,lentoken,gendic)           
                if(not len(compressed)):
                    debugw("unknown word in gendic sequence, aborting")
                    compressed = bytearray()
                    return compressed
    # dump whole compressed stream
    debugw("compressed ngram is=")
    debugw(compressed.hex())
    debugw("compressed ngram byte length is=")
    debugw(len(compressed))

    return compressed

def compress_second_pass(compressed):

    ngram_compressed = bytearray()
    ngram_length = 0
    ngram_byte_length = 0
    index_jumps = []
    candidates = []
    idx = 0
    # second pass main loop
    #debugw("compressed=")
    #debugw(compressed)
    while (idx < len(compressed)):

        debugw("second pass idx=")
        debugw(idx)
        idxchar = 0
        reset_ngram = False
        debugw("indexjumps=")
        debugw(index_jumps)


        if(not (compressed[idx] & 128)):
            ngram_compressed.append(compressed[idx])
            debugw("".join([f"\\x{byte:02x}" for byte in ngram_compressed]))
            debugw("super common ext")
            idx += 1
            index_jumps.append(1)
            ngram_byte_length += 1
        elif((compressed[idx] & 128) and (not (compressed[idx+1] & 128))):
            ngram_compressed.extend(compressed[idx:idx+2])
            debugw("".join([f"\\x{byte:02x}" for byte in ngram_compressed]))
            debugw("common ext")
            idx += 2
            index_jumps.append(2)
            ngram_byte_length += 2
        elif((compressed[idx] & 128) and (compressed[idx+1] & 128) and (not compressed[idx+2] & 128)):
            ngram_compressed.extend(compressed[idx:idx+3]) 
            debugw("".join([f"\\x{byte:02x}" for byte in ngram_compressed]))
            debugw("rare ext")
            idx += 3  
            index_jumps.append(3)
            ngram_byte_length += 3     
        elif((compressed[idx] == 255) and (compressed[idx+1] == 255) and (compressed[idx+2] == 255)):
            # TODO : take into account 4 escape sequences instead of only one.
            #reset ngram_compressed
            char = compressed[idx+3]
            debugw("unknown token sequence detected")
            #print(char)
            #str = ""
            idxchar = 0
            while(char != 0):
                   idxchar += 1
                   char = compressed[idx+3+idxchar]
                   debugw("char=")
                   debugw(char)
            debugw("end of unknown token sequence detected at idx:")
            idx += (3 + idxchar)
            debugw(idx)
            index_jumps.append(3 + idxchar)
            ngram_length -= 1
            reset_ngram = True
         
        elif((compressed[idx] & 128) and (compressed[idx+1] & 128) and (compressed[idx+2] & 128)):
            # Session DIC space, breaks ngram construction.
            debugw("session DIC space, we break ngram construction")
            idx += 3
            index_jumps.append(3)
            ngram_length -= 1
            reset_ngram = True
    

        ngram_length += 1
        debugw("indexjumps=")
        debugw(index_jumps)
        debugw("ngram_length")
        debugw(ngram_length)

        if (((ngram_length == 3) and (ngram_byte_length > 3)) or (ngram_length == 4)):
            # if there are contractions, apparent ngram length will be one token less and potentially present in N4 ngrams
            # try to replace the ngram if it exists, and only if ngram_byte_length is > 3, otherwise there will be no compression gain.
            # save index jumps for rewind operations.
            # TO BE CONTINUED .....
            try: 
                
                ngram_compressed_no_ascii = "".join([f"\\x{byte:02x}" for byte in ngram_compressed])
                ngram_compressed_no_ascii = ngram_compressed_no_ascii.replace("\\","")
                debugw(ngram_compressed_no_ascii)
                code = ngram_dict[ngram_compressed_no_ascii]
                debugw("****FOUND*****")
                ratio = ngram_byte_length/3 # all ngrams are encoded in a 3 byte address space, hence div by 3
                removebytes = ngram_byte_length
                if(idxchar):
                    insertpos = idx - ngram_byte_length - (3 + idxchar)
                else:
                    insertpos = idx - ngram_byte_length                
                candidates.append((code,insertpos,removebytes,ratio))
            except:
                #traceback.print_exc()
                debugw("no luck 3N/4N")

            # reset all ngram data
            ngram_length = 0
            ngram_byte_length = 0
            ratio = 0
            removebytes = 0
            ngram_compressed = bytearray()

            #rewind...and retry a new ngram window from initial token index + one token shift
            #BUG HERE !!
            debugw("indexjumps=")
            debugw(index_jumps)
            #time.sleep(0.1)
            debugw("lastindexjumps_except_first=")
            debugw(index_jumps[-len(index_jumps)+1:])
            debugw("index_before_rewind=")
            debugw(idx)

            idx -= sum(index_jumps[-len(index_jumps)+1:])
            index_jumps = []
            debugw("idx after rewind=")
            debugw(idx)

        elif (reset_ngram):
            debugw("ngram reset : unknown token starts before ngram_length 3 or 4")
            ngram_length = 0
            ngram_byte_length = 0
            ratio = 0
            removebytes = 0
            #do not rewind : reset pos after unknown sequence
            index_jumps = []

    return candidates        


def process_candidates_v2(candidates):

    #here we scan all candidates.
    #if there are overlaps, we select the candidate with the best ratio, if any.
    #The result is a reduced list of candidates data.

    #Next we recreate the compressed stream and replace the bytes at insertpos by the candidate code
    debugw(candidates)
    candidates_reduced = []
    idx_reduced = 0
    idx = 0
    deleted_candidates_number = 0

    mutual_overlaps = []
    overlap_idx = 0

    while(idx < len(candidates)):
        
        code = candidates[idx][0]
        insertpos = candidates[idx][1]
        removebytes = candidates[idx][2]
        ratio = candidates[idx][3]

        first_overlap = True
        
        for idx_lookahead in range(idx+1,len(candidates)):
            
            code_lookahead = candidates[idx_lookahead][0]
            insertpos_lookahead = candidates[idx_lookahead][1]
            removebytes_lookahead = candidates[idx_lookahead][2]
            ratio_lookahead = candidates[idx_lookahead][3]

            if((insertpos + removebytes - 1) >= insertpos_lookahead):
                
                debugw("overlap!")
                debugw(code)
                debugw(code_lookahead)
                
                #add mutually overlapping indexes to an array
                if(first_overlap):
                    mutual_overlaps.append([idx])
                    mutual_overlaps[overlap_idx].append(idx_lookahead)
                    first_overlap = False

                else:
                    # case for a mutual overlap of at least 3 ngrams
                    debugw("len mutual overlap:")
                    debugw(len(mutual_overlaps))
                    debugw("overlap_idx")
                    debugw(overlap_idx)
                    mutual_overlaps[overlap_idx].append(idx_lookahead)
                 
                    overlap_idx += 1
                
            else:
                #end of mutual overlap (current lookahead is not overlapping with original idx)
                break
        idx += 1        
    #keep best ratio from all overlap lists
    keep_idxs = []
    remove_idx_shift = 0
        
    for overlap in mutual_overlaps:

        prev_candidate_ratio = 0
        
        for candidate_idx in overlap:

            debugw("candidate_idx:")
            debugw(candidate_idx)
            candidate_ratio = candidates[candidate_idx - remove_idx_shift][3]
            if (candidate_ratio >= prev_candidate_ratio):
                keep_idx = candidate_idx
                prev_candidate_ratio = candidate_ratio

        keep_idxs.append(keep_idx)

        

        for candidate_idx in overlap:
            if(candidate_idx != keep_idx):
                debugw("candidate len:")
                debugw(len(candidates))
                
                debugw("will delete idx:")
                debugw(str(candidate_idx - remove_idx_shift))
                
                del candidates[candidate_idx - remove_idx_shift]
                deleted_candidates_number += 1
                debugw("deleted idx:")
                debugw(str(candidate_idx - remove_idx_shift))
                remove_idx_shift += 1
                #keep the best ratio only from the list of mutual overlaps

    if (deleted_candidates_number > 0):
        debugw("recursive")
        deleted_candidates_number = 0
        process_candidates_v2(candidates)

    #need to exit recursion when len candidates stops decreasing

    return candidates

def ngram_insert_reserved_bits(ngram_compressed):
            
    debugw("".join([f"\\x{byte:02x}" for byte in ngram_compressed]))

    #convert to bitarray
    c = bitarray(endian='little')
    c.frombytes(ngram_compressed)
    debugw(c)
    
    # set msb of first byte to 1 and shift the bits above up.
    c.insert(7,1)
    debugw(c)

    # set msb of second byte to 1 and shift the bits above up.
    c.insert(15,1)
    debugw(c)

    # remove two excess bits that arose from our shifts
    del c[24:26:1]
    debugw(c)
    
    # replace the original ngram_compressed bytearray with our tweaked bytes
    ngram_compressed = bytearray()
    ngram_compressed.append((c.tobytes())[0])
    ngram_compressed.append((c.tobytes())[1])
    ngram_compressed.append((c.tobytes())[2])

    return ngram_compressed
                

def replace_candidates_in_processed(candidates,processed):

    byteshift = 0
    shiftcode = 0
    for candidate in candidates:
            insertpos = candidate[1] - byteshift
            removebytes = candidate[2]
            del processed[insertpos:insertpos + removebytes]
            byteshift += removebytes
            ## first we need to convert candidate code to proper 3 byte format
            # we add our 4 ngram code space at a 2^20 shift in the 3 bytes address space. 
            shifted_code = 524416 + candidate[0]
            # now we convert our shifted ngram code to a byte sequence in the compressed format
            bytes_shiftedcode = shifted_code.to_bytes(3, byteorder='little')
            # print it
            debugw(bytes_shiftedcode)
            # tweak the bytes to insert reserved bits for 1/2/3 bytes variable length encoding
            # compliance.
            bytes_shiftedcode = ngram_insert_reserved_bits(bytes_shiftedcode)
            # print it
            debugw(bytes_shiftedcode)
            # now we insert it at the position of the non-compressed ngram
            processed[insertpos:insertpos] = bytes_shiftedcode
            # we added 3 bytes, we have to compensate to keep future insertpos valid.
            byteshift -= 3

    return processed


def ngram_process_rules(subtokens):

    ### VARIOUS DETOKENIZER CLEANUP/FORMATTING OPERATIONS
    processed_ngram_string = ""
    capitalize = False
    token_idx = 0
    for token in subtokens:

        if(capitalize):
            token = token.capitalize()
            capitalize = False

        # English syntactic rules : remove whitespace left of "!?." 
        # and enforce capitalization on first non whitespace character following.
        if (re.match("[!\?\.]",token)):
            processed_ngram_string += token
            capitalize = True

        # English syntactic rules : remove whitespace left of ",;:" 
        elif (re.match("[,;:]",token)):         
            processed_ngram_string += token
            capitalize = False

        # append whitespace left of added token
        else:
            processed_ngram_string = processed_ngram_string + " " + token

        token_idx += 1
        
        if(len(subtokens) == token_idx):
            debugw("last token of ngram")
            processed_ngram_string += " "

    return processed_ngram_string

def decompress_ngram_bytes(compressed):

    idx = 0
    detokenizer_ngram = []
    
    while(idx < len(compressed)):
    
        if(not (compressed[idx] & 128)):
            
            # current index byte msb is at 0, 
            # it is one of the 128 first tokens in the dictionary.
            debugw("super common word")
            #decode in place
            
            inta = compressed[idx]        
            detokenizer_ngram.append(engdict[inta])
            idx += 1

        elif((compressed[idx] & 128) and (not (compressed[idx+1] & 128))):

            # current index byte msb is at 1, and next byte msb is at 0. 
            # it is one of the 16384 next tokens in the dictionary.
            debugw("common word")

            # populate bitarray from the two bytes
            c = bitarray(endian='little')
            c.frombytes(compressed[idx:idx+2])
            debugw(c)

            # remove first byte msb (shift down the bits above)
            del c[7]
            debugw(c)

            # convert bytes array to 16 bit unsigned integer
            inta = (struct.unpack("<H", c.tobytes()))[0]
            # add offset back so we get a valid dictionary key
            inta += 128

            # print word
            detokenizer_ngram.append(engdict[inta])
            # increment byte counter with step 2, we processed 2 bytes.
            idx += 2

        #elif((compressed[idx] & 128) and (compressed[idx+1] & 128)):
        elif((compressed[idx] & 128) and (compressed[idx+1] & 128) and (not compressed[idx+2] & 128)):
            
            # current index byte msb is at 1, and next byte mbs is at 1. 
            # it is one of the 4194304 next tokens in the dictionary.
            debugw("rare word")
            
            chunk = compressed[idx:idx+3]

            # populate bitarray from the three bytes
            c = bitarray(endian='little')
            #c.frombytes(compressed[idx:idx+3])
            c.frombytes(chunk)
            
            debugw(c)

            # remove second byte msb (shift down the bits above)
            del c[15]
            debugw(c)

            # remove first byte msb (shift down the bits above)
            del c[7]
            debugw(c)

            c.extend("0000000000") 
            # pad to 4 bytes (32 bit integer format) : 3 bytes + 10 bits 
            # because we previously removed two bits with del c[15] and del c[7]
            debugw(c)

            # convert bytes array to 32 bit unsigned integer
            inta = (struct.unpack("<L", c.tobytes()))[0]

            inta += (16384 + 128)

            detokenizer_ngram.append(engdict[inta])

            # increment byte counter with step 3, we processed 3 bytes.
            idx += 3

    return detokenizer_ngram


###INLINE START###

#downloading tokenizer model if missing
nltk.download('punkt')

#opening the english dict of most used 1/3 million words from google corpus of 1 trillion words.
#special characters have been added with their respective prevalence (from wikipedia corpus)
#contractions also have been added in their form with a quote just after (next line) the form 
# without quote. ex : next line after "dont" appears "don't"

file1 = open('count_1w.txt', 'r')
Lines = file1.readlines()

#initializing Python dicts
count = 1
engdict = {}
engdictrev = {}


# special case : byte val 0 is equal to new line.
# TODO : make sure that windows CRLF is taken care of.
engdict[0] = "\n"
engdictrev["\n"] = 0

# populating dicts
for line in Lines:
    # Strips the newline character
    engdict[count] = line.strip()
    engdictrev[line.strip()] = count
    count += 1

### populating ngram dict

filengrams = open('outngrams.bin', 'rt')
ngramlines = filengrams.readlines()

ngram_dict = {}
ngram_dict_rev = {}


count = 0
# populating dicts
for ngramline in ngramlines:
# Strips the newline character
    #keystr = "".join([f"\\x{byte:02x}" for byte in ngramline.strip()])
    #keystr = keystr.replace("\\","")
    #if(count == 71374):
    keystr = ngramline.strip()
    #print(ngramline.strip())
    #print(keystr)
    #quit()
    ngram_dict_rev[count] = keystr
    ngram_dict[keystr] = count
    count += 1

idx = 0
debugw("first ngram in dict:")
test = ngram_dict_rev[0]
debugw(test)
debugw(ngram_dict[test])
count = 0


if (compress):

    tokens = []
    # check if file is utf-8
    if(check_file_is_utf8(infile)):
        with codecs.open(infile, 'r', encoding='utf-8') as utf8_file:
            # Read the content of the UTF-8 file and transcode it to ASCII
            # encode('ascii','ignore') MAY replace unknown char with chr(0)
            # We don't want that, as it is a termination char for unknown strings.
            # on the other hand backslashreplace replaces too much chars that could be transcribed
            # the best option for now it check for chr(0) presence before writing the unknown token representation.
            ascii_content = utf8_file.read().encode('ascii', 'ignore').decode('ascii')
            #debugw(ascii_content)
            Linesin = ascii_content.splitlines()
            if(debug_on):
                outfile_ascii = infile + ".asc"
                with codecs.open(outfile_ascii, "w", encoding='ascii') as ascii_file:
                    ascii_file.write(ascii_content)
    else:
        # Reading file to be compressed
        file2 = open(infile,'r')
        #text = file2.read()
        Linesin = file2.readlines()

    if(gendic):
         if(len(outfile)):
                fh = open(outfile, 'wt')

    lineidx = 0
    for line in Linesin:
        line = line.lower()

        # First pass tokenizer (does not split adjunct special chars)
        line_tokens = tknzr.tokenize(line)
        #debugw(line_tokens)

        if( not gendic):
            tokens.append(line_tokens)
        else:
            compressed = compress_tokens(line_tokens,gendic)
            if(len(outfile) and len(compressed)):
                # write compressed binary stream to file if supplied in args or to stdout otherwise.
                hexstr = "".join([f"\\x{byte:02x}" for byte in compressed])
                hexstr = hexstr.replace("\\","")
                fh.write(hexstr)
                if(debug_ngrams_dic):
                    fh.write("\t")
                    strline = str(lineidx)
                    fh.write(strline)
                fh.write("\n")
            else:
                sys.stdout.buffer.write(compressed)
                sys.stdout.buffer.write(b"\n")
        lineidx += 1
    #line_tokens.append("\n")
    #tokens = tokens + line_tokens
    debugw(tokens)
    
    if (not gendic):

        compressed = compress_tokens(tokens,gendic)

        if(secondpass):
            candidates = compress_second_pass(compressed)
            debugw("candidates:")
            debugw(candidates)
            processed_candidates = process_candidates_v2(candidates)
            debugw("processed candidates:")
            debugw(processed_candidates)
            compressed = replace_candidates_in_processed(processed_candidates,compressed)


        # write compressed binary stream to file if supplied in args or to stdout otherwise.
        if(len(outfile)):
            with open(outfile, 'wb') as fh:
                fh.write(compressed)
        else:
            sys.stdout.buffer.write(compressed)

        for sessidx in range(2113664,unknown_token_idx):
            debugw("session_index:" + str(sessidx))
            debugw(engdict[sessidx])
            debugw(engdictrev[engdict[sessidx]])
            debugw("session_index:" + str(sessidx))

    fh.close()

# decompress mode
else:

    # decoding part
    debugw("decoding...")
    detokenizer = []
    detokenizer_idx = 0

    if(len(infile)):
        with open(infile, 'rb') as fh:
            compressed = bytearray(fh.read())

    idx = 0
    #FirstCharOfLine = 1
    CharIsUpperCase = 1
    #CharIsUpperCase2 = 0
    
    # main decoding loop
    while (idx < len(compressed)):
            
            # write each byte
            debugw(hex(compressed[idx]))

            #if( (idx > 0) and compressed[idx] == 0 and compressed[idx - 1] == 0):
            #find len of consecutive 0 chars

            if(idx < len(compressed) -1):
                if((compressed[idx] == 0) and (compressed[idx+1] != 0)):
                    #FirstCharOfLine = 1
                    CharIsUpperCase = 1
                elif(CharIsUpperCase == 1):
                    #FirstCharOfLine = 2
                    CharIsUpperCase = 2
                        
            if(len(detokenizer) > 0):


                ### VARIOUS DETOKENIZER CLEANUP/FORMATTING OPERATIONS

                #ensure this is not the end of an ngram. ngrams necessarily contain whitespaces
                if (not re.search(" ",detokenizer[detokenizer_idx-2])):
                    # English syntactic rules : remove whitespace left of "!?." 
                    # and enforce capitalization on first non whitespace character following.
                    if (re.match("[!\?\.]",detokenizer[detokenizer_idx-2]) and detokenizer_idx > 2):
                        del detokenizer[detokenizer_idx-3]
                        detokenizer_idx -= 1
                        if(CharIsUpperCase != 1):
                            CharIsUpperCase = 2

                    # English syntactic rules : remove whitespace left of ",;:" 
                    if (re.match("[,;:]",detokenizer[detokenizer_idx-2]) and detokenizer_idx > 2):         
                        del detokenizer[detokenizer_idx-3]
                        detokenizer_idx -= 1

                    # URL/URI detected, remove any spurious whitespace before "//" 
                    if (re.match("^\/\/",detokenizer[detokenizer_idx-2]) and detokenizer_idx > 2):         
                        del detokenizer[detokenizer_idx-3]
                        detokenizer_idx -= 1
                    
                    # E-mail detected, remove whitespaces left and right of "@"
                    if (re.match("@",detokenizer[detokenizer_idx-2]) and detokenizer_idx > 2):         
                        del detokenizer[detokenizer_idx-3]
                        detokenizer_idx -= 1
                        del detokenizer[detokenizer_idx-1]
                        detokenizer_idx -= 1

            if(not (compressed[idx] & 128)):
                
                # current index byte msb is at 0, 
                # it is one of the 128 first tokens in the dictionary.
                debugw("super common word")
                #decode in place
                
                inta = compressed[idx]
                       
                if(CharIsUpperCase == 2):
                    detokenizer.append(engdict[inta].capitalize())
                    detokenizer_idx += 1
                    CharIsUpperCase = 0
                else:    
                    detokenizer.append(engdict[inta])
                    detokenizer_idx += 1
                  
                # print to stdout
                if(CharIsUpperCase != 1):
                    detokenizer.append(" ")
                    detokenizer_idx += 1

                debugw(engdict[inta])
                idx += 1

            elif((compressed[idx] & 128) and (not (compressed[idx+1] & 128))):
    
                # current index byte msb is at 1, and next byte msb is at 0. 
                # it is one of the 16384 next tokens in the dictionary.
                debugw("common word")
    
                # populate bitarray from the two bytes
                c = bitarray(endian='little')
                c.frombytes(compressed[idx:idx+2])
                debugw(c)
    
                # remove first byte msb (shift down the bits above)
                del c[7]
                debugw(c)

                # convert bytes array to 16 bit unsigned integer
                inta = (struct.unpack("<H", c.tobytes()))[0]
                # add offset back so we get a valid dictionary key
                inta += 128
    
                # print word
                if(CharIsUpperCase == 2):
                    detokenizer.append(engdict[inta].capitalize())
                    detokenizer_idx += 1
                    CharIsUpperCase = 0
                else:
                    detokenizer.append(engdict[inta])
                    detokenizer_idx += 1   

                if(CharIsUpperCase != 1):
                    detokenizer.append(" ")
                    detokenizer_idx += 1 
                
                debugw(engdict[inta])
                # increment byte counter with step 2, we processed 2 bytes.
                idx += 2
    
            #elif((compressed[idx] & 128) and (compressed[idx+1] & 128)):
            elif((compressed[idx] & 128) and (compressed[idx+1] & 128) and (not compressed[idx+2] & 128)):
                
                # current index byte msb is at 1, and next byte mbs is at 1. 
                # it is one of the 4194304 next tokens in the dictionary.
                debugw("rare word")
                
                chunk = compressed[idx:idx+3]

                # populate bitarray from the three bytes
                c = bitarray(endian='little')
                #c.frombytes(compressed[idx:idx+3])
                c.frombytes(chunk)
                
                debugw(c)

                # remove second byte msb (shift down the bits above)
                del c[15]
                debugw(c)

                # remove first byte msb (shift down the bits above)
                del c[7]
                debugw(c)

                c.extend("0000000000") 
                # pad to 4 bytes (32 bit integer format) : 3 bytes + 10 bits 
                # because we previously removed two bits with del c[15] and del c[7]
                debugw(c)

                # convert bytes array to 32 bit unsigned integer
                inta = (struct.unpack("<L", c.tobytes()))[0]

                if (inta >= 524416):
                    # this is a ngram.
                    # remove offset to get into ngram dic code range.
                    inta -= 524416
                    debugw("this is an ngram. code:")
                    debugw(inta)
                    # process ngram through ngram dictionary
                    # replace ngram code with corresponding ngram string and add them to the tokenizer
                    ngram_string = ngram_dict_rev[inta]
                    debugw("ngram string:")
                    debugw(ngram_string)
                    subs = 0
                    #(ngram_string,subs) = re.subn(r'x',r'\\x',ngram_string)
                    (ngram_string,subs) = re.subn(r'x',r'',ngram_string)   
                    debugw("ngram string:")
                    debugw(ngram_string)
                    ngram_bytes = bytes.fromhex(ngram_string)
                    subtokens = decompress_ngram_bytes(ngram_bytes)
                    #bytes = bytearray(ngram_string,encoding="ascii")
                    #subtokens.insert(0,"PREFIX")
                    #subtokens.append("SUFFIX")
                    
                    
                    #subtokens = nltk.word_tokenize(ngram_string)
                    # We know there shouldn't be any new lines in the subtokens.
                    # possessives, contractions or punctuation may occur.
                    # we need to add capitalization rules and spaces after punctuation rules.
                    # These should be catched by the detokenizer backward processor (detokenizer_idx -2)
                    # The problem is we append more than one token.
                    # So we should process rules for first subtoken insertion only.
                    # The rest should have inline processing (here)

                    if(CharIsUpperCase == 2):
                        detokenizer.append(subtokens[0].capitalize())
                        detokenizer_idx += 1
                        CharIsUpperCase = 0
                    else:
                        detokenizer.append(subtokens[0])
                        detokenizer_idx += 1 
                    #if(CharIsUpperCase != 1):
                    #    detokenizer.append(" ") 
                    #    detokenizer_idx += 1

                    ngram_processed_string = ngram_process_rules(subtokens[1:])
                    # We shoud take care that the backward detokenizer processor does not mingle
                    # with the the rest of the ngram string.
                    # Such a special token will be the only one to have whitespaces in it
                    # So we can detect it this way
                    detokenizer.append(ngram_processed_string)
                    detokenizer_idx += 1
                                        

                else:
                    inta += (16384 + 128)

                    if(CharIsUpperCase == 2):
                        detokenizer.append(engdict[inta].capitalize())
                        detokenizer_idx += 1
                        CharIsUpperCase = 0
                    else:
                        detokenizer.append(engdict[inta])
                        detokenizer_idx += 1 
                    if(CharIsUpperCase != 1):
                        detokenizer.append(" ") 
                        detokenizer_idx += 1
                    
                    debugw(engdict[inta])
                    # increment byte counter with step 3, we processed 3 bytes.
                idx += 3

            #elif((compressed[idx] == 255) and (compressed[idx+1] == 255) and (compressed[idx+2] == 255)):   
            elif((compressed[idx] & 128) and (compressed[idx+1] & 128) and (compressed[idx+2] & 128)):
            
                #check if Huffmann first

                chunk = compressed[idx:idx+3]

                # populate bitarray from the three bytes
                c = bitarray(endian='little')
                #c.frombytes(compressed[idx:idx+3])
                c.frombytes(chunk)
                
                debugw(c)

                # remove third byte msb (shift down the bits above)
                del c[23]
                debugw(c)

                # remove second byte msb (shift down the bits above)
                del c[15]
                debugw(c)

                # remove first byte msb (shift down the bits above)
                del c[7]
                debugw(c)

                c.extend("00000000000") 
                # pad to 4 bytes (32 bit integer format) : 3 bytes + 8 bits + 3 bits 
                # because we previously removed three bits with del c[23], del c[15] and del c[7]
                debugw(c)

                # convert bytes array to 32 bit unsigned integer
                inta = (struct.unpack("<L", c.tobytes()))[0]
                inta -= 2097151
                # if it is a Huffmann select tree code it will be 0 to 4 included
                # if it is a session DIC it will be shifted in the negatives.

                if (inta in range(0,5)):        

                    # unknown word
                    # end check if Huffmann first
                    debugw("unknown word escape sequence detected, code: " + str(inta))
                    #unknown word escape sequence detected.
                    if(inta == 0):
                        char = compressed[idx+3]
                        stra = ""
                        idxchar = 0
                        while(char != 0):
                            debugw("char=")
                            debugw(char)
                            stra += chr(char)
                            debugw("printing string state=")
                            debugw(stra)
                            idxchar += 1
                            char = compressed[idx+3 + idxchar]
                        debugw("termination char detected=")
                        debugw(char)
                    else:
                        bstr = bytearray()
                        idxchar = 0
                        while(char != 0):
                            bstr.append(char)
                            idxchar += 1
                            char = compressed[idx+3 + idxchar]
                        debugw("huffmann : termination char detected=")
                        debugw(char)
                        stra = decode_unknown(bstr,inta)
                        #stra = codec.decode(bstr)    
                    
                    debugw("we append that unknown word in our session dic at idx: " + str(unknown_token_idx) + " since it may be recalled")
                    engdictrev[stra] = unknown_token_idx
                    engdict[unknown_token_idx] = stra
                    unknown_token_idx += 1
                    
                        
                    if(CharIsUpperCase == 2):
                        detokenizer.append(stra.capitalize())
                        detokenizer_idx += 1
                        CharIsUpperCase = 0
                    else:
                        detokenizer.append(stra)
                        detokenizer_idx += 1 
                    if(CharIsUpperCase != 1):
                        detokenizer.append(" ") 
                        detokenizer_idx += 1
    
                else:

                    inta += 2097151
                    # it is a session DIC, shifting back to 0.
                    inta += (2097152 + 16384 + 128)
                    # it is a session DIC, shifting back session dic address space.

                    debugw("recalled word:")
                    
                    try:
                        debugw(engdict[inta])
                        # print word
                    
                        if(CharIsUpperCase == 2):
                            detokenizer.append(engdict[inta].capitalize())
                            detokenizer_idx += 1
                            CharIsUpperCase = 0
                        else:
                            detokenizer.append(engdict[inta])
                            detokenizer_idx += 1   

                        if(CharIsUpperCase != 1):
                            detokenizer.append(" ")
                            detokenizer_idx += 1 
                    
                    except:
                        debugw("something went wrong, could not find word in session DIC")

                        for sessidx in range(2113664,unknown_token_idx):
                            debugw("session_index:" + str(sessidx))
                            debugw(engdict[sessidx])
                            debugw(engdictrev[engdict[sessidx]])
                            debugw("session_index:" + str(sessidx))


                idx += 3 + idxchar

    debugw(detokenizer)
    if not(len(outfile)):
        print(''.join(detokenizer))
    else:
        # write clear text to file if supplied in args
        with open(outfile, 'w') as fh:
            fh.write(''.join(detokenizer))
    

Audio shelving filter with resonance.

This shelving filter was obtained serendipitously by combining an op-amp integrator stage (bottom) with an all-pass filter stage (top). The top stage output is the filter output.

I have not done the analytical determination of the transfer function down to the resistance and capacitance of each component. If you wish to do so, This could be done either through FACT – fast analytical circuit techniques, or automatically with some Ltspice adjunct program like SLICAP.

Thus the behavior was mostly determined empirically by sweeping all resistances and capacitances, with resistances replaced by potentiometers.

It exhibits resonance. Most shelving filters do not exhibit this feature. It may be desired in some cases.

It can also switch between a high shelf and a low shelf behavior.

Control is a bit touchy as the filter shows a certain degree of inter-dependence with respect to some potentiometer effects. (Certain potentiometers affect several characteristics at once)

Schematic
wiper_fb sweep – filter output is top graph
wiper_LP_gain sweep – filter output is top graph

It requires the TL072 model for the op-amps.

There is a wave input if you wish to test it on a wave file. The filename is input.wav and the output filename is output.wav. Connect the “in” net instead of the V5 source and change the simulation to .tran

Don’t forget to set the transient analysis to the length of the input wave file.

The zip below contains the ASC file and some screenshots of the frequency response while stepping some potentiometers.

Have fun!

DOWNLOAD :

Ltspice synchronous generator model in abc reference frame coupled with VAWT prime mover and DC battery charging load.

Simulation model

Two simulation models are provided :

VAWT_generator_and_crowbar.asc : Provides the VAWT generator model and synchronous generator with excitation winding, rectifier, and crowbar circuit.

VAWT_generator_and_PWM_charger.asc : Provides the VAWT generaor model and synchronous generator wih excitation winding, rectifier, and crude PWM charger. (for 48V battery bank).

Limitations are mainly due to the simulation speed restrictions.

Requires Ltspice 17.1 or newer. May require some component models, such as for specific MOSFETS.

Incentive.

The following work proposes an exploration of behaviour of medium size residential/agro/community wind turbines when coupled to a synchronous generator with field excitation.

Traditionnally, small synchronous generators with field excitation are usually found in small hydroelectric generation setups.

While most commercial devices in the wing generation category are fitted with PMSM (Permanent Magnet Synchronous Generators).

PMSM usually present lower inertia, include rare earth material magnets which contribute significantly to the overall cost of the generator, and which cost is expected to grow significantly due to undersupply and large demand for the EV market.

https://www.mining.com/supply-of-rare-earth-magnets-wont-keep-with-demand-by-2040-report/

Permanent Magnets are desirable for small to medium power sizes due to the absence of field losses, which contribute substantially to the total power loss of the generator for small scales, Beside magnets most PMSM rotor configuration use electrical steel to form either salient or round rotors. The main difference from an economical perspective is cost of copper or aluminum vs rare earth magnets as initial cost, and lower efficiency due to field excitation as a variable and recurring loss. These range in the 2% to 4% for operation at nominal conditions for a 75kVA machine. Our investigation will assume first a constant excitation, variable voltage operation, While control of excitation will be considered for dump load operation (braking) on a static resistive load, with the intent of thermal generation in high wind power conditions. Also, we will take into account the stalling speed of the VAWT and update the control law accordingly to prevent this phenomenon. We will also investigate alternative control laws for dump load operation and braking, under constant field and variable output impedance (simulating a servo-rheostat load). These are mainly of use for PMSM dump load operation (whose field is constant)

Synchronous Generator model part.

The model represents a synchronous machine with the following characteristics :

3 phase stator, wye configuration.

Variable field excitation, though the model does not implement AVR based on excitation control, at least for now. Variable field excitation is investigated for dump load (crowbar) operation.

Note that permanent magnet synchronous generator could be crudely modelled by this instance by setting an adequate static field strength (with no ramp up) and by setting a higher number of poles. Note that most PMSG designs do not use dampers.

Parametric number of poles. Note also that q-damper winding inductance is valid for a two pole machine, for p > 2, q-damper self and mutual inductances should be set to 0. (or the q-damper excluded from the circuit)

Rotor Saliency is modelled, through the Lms parameter.

Model is based on flux linkages. It specifies leakage, mutual inductance, and resistance of each winding. The model is ‘lumped’ in terms of inductances, i.e. it does not specify inductances as a function of winding geometry (number of turns, area, length, permeability.. etc.)

All mutual inductances are modelled (stator phase to each other stator phase, rotor field to each rotor damper, and stator phase to each rotor winding)

The default setup includes one damper coil on the d axis and one damper coil on the q axis, and uses a two pole rotor by default.

This makes 6 flux expressions with 6 components each : 1 flux component arising from self inductance, and 5 components corresponding to mutual inductances since there are 6 windings in total.

Due to a quirk in the Ltspice parser for arbitrary inductors based on flux expressions, I had to separate self flux from the mutual flux expressions.

Ltspice expects the current flowing through the inductor to be represented by the variable ‘x’. Thus I had to use the form Flux = Lself*x + sum(flux_linkages)

That is why the flux linkages expressions in the following screenshots have 5 components instead of 6. the final flux expression combining them all is specified in the inductor ‘Flux=’ expression, seen in Figure 5.

Figure 1 Natural inductances parameters
Figure 2 Stator self inductances expressions
Figure 3 Stator flux linkages expressions
Figure 4 Rotor flux linkages expressions
Figure 5 Stator equivalent circuit
Figure 6 Rotor equivalent circuits

Magnetic saturation effects are not modelled for now. Usually the field winding is driven close to saturation, which makes this modelling significant, at least for large generators.

Space Flux distribution does not model anistropies coming from the rotor shoe and stator wedges and slots physical geometry. These flux anisotropies give rise to EMF with harmonic distortion. It only models those coming from the saliency model, which is a first order approximation.

The model uses the LTspice arbitrary inductor model to express self flux and flux linkages. The windings thus use inductors as the source of emf, not behavioural voltage sources. The only inductor that is powered by a DC source is the field winding.

The main incentive of using abc frame equivalent circuit instead of a faster dq0 reference is that :

A model in abc reference frame has the advantage of being better suited for non steady state, non linear loads, islanded mode (not connected to the grid).

As an example, the model feeds a resistive load and smoothing capacitor through a 6 diode three phase passive rectifier. Despite the load non linearity, the model performs well.

The model does not drive the shaft at synchronous speed (steady state turns/min shaft rotational speed provided by the manufacturer), it takes as an input mechanical power from the VAWT, which itself is a function of wind speed (provided through the V12 source PWL input) and VAWT rotor speed. A steady mechanical input power, modelled through the V2 source, can be swapped instead of the VAWT for debugging purposes.

A further refinement of the model for MPPT modelling could make use of a WAV file input as the source of real world wind speed data to model gust surges.

Mechanical losses are modelled through friction and windage power losses of the generator assembly, which are assumed to be constant.

Inductance and resistance parameters.

We used natural SI units, not the p.u. system.

The main challenge with such a model in abc reference frame is that manufacturers specify alternator parameters as synchronous reactances, transient and subtransient reactances, machine time constants, and often in p.u units. Alternators are not designed per se to operate at electrical frequencies other than 50 or 60 Hz, and for larger models, these are also intended to be grid tied, so it follows that manufacturers provide parameters related to their intended use and standardized for the use of industry specific simulation software. ( for the larger, >1MVA models)

That means that, these have to be converted back to natural self and mutual inductances. Conversion to SI units from p.u. is straightforward, as well as reactances to inductances.

The real challenge is to derive abc reference ‘natural’ inductances from the previously obtained inductances.

Some might be measured experimentally, such as resistances, if the generator is available at hand. As for inductances, it is specially hard for dampers which are shorted and without external leads. For the other windings, it is also hard because inductances vary as a function of rotor position. One could get an approximation of self inductances by measuring one winding while all the other are shorted, and mutual inductances by measuring one winding and the mutual winding under investigation, while the others are shorted, and repeat these measurements under various rotor angles, to determine inductances minima and maxima. But damper windings inductances would still not be measurable.

The proposed method is explained here :

https://electronics.stackexchange.com/questions/678068/3-phase-synchronous-machine-with-salient-rotor-inductances-and-resistances-measu

It assumes that the stator is star (wye) wired and that the neutral point is accessible. The problem with the method is the frequency of the measurement by an LCR meter, often above 100kHz, which makes rise to skin effect that affects inductance measurement by lowering the measured value vs. reality, as well as the low current used in measurement which make it happen in a portion of the B/H curve which is not the one of nominal currents. Since electrical steel B/H curve is not fully linear, (permeability is not a constant factor) and lower at very low currents, it has the effect of also lowering the measured inductance. How that affects all the inductances ratios is a whole other issue. This method is probably more accurate for low number of poles, ideally two. Whether the proposed method has any practical and theoretical validity is not certain, so take it with a grain of salt. I personally could not find any resource that performed the experiment. A selsyn synchro, with a two pole rotor and three phase stator or a wye wired car alternator with accessible neutral and removed AVR and rectifier diodes could be put to the test bench with this method. Note that a car alternator claw pole rotor arrangement is not faithfully represented by this Ltspice model.

Usually, most experimental methods deal with the determination of synchronous reactances Xd, Xq.

The Electric Generators Handbook from Ion Boldea explains the method in chapter 4.

Theoretical determination of natural mutual inductances from datasheet reactances (synchronous, transient, subtransient)

The following paper propose a method to determine mutual inductances from datasheet parameters, it makes use of scaling parameters Kf,Kd,Kq to make the conversion, which yields an approximation :

(1)

https://studylib.net/doc/18754307/analysis-of-synchronous-machine-modeling-for-simulation-and

Analysis of synchronous machine modeling for simulation
and industrial applications
Barakat,Tnani,Champenois,Mouni

Issues with the model with the default parameters

Note that the 75kVA generator parameters used in the present Ltspice models are derived from the paper (1). Excitation field winding resistance specified in the paper (around 2 ohms) gives unbearably high field losses in comparison to power output at low prime mover input power, So it has been lowered. A field resistance of about 2 ohms is common for machines in that power range.

This methodology is questionable, but it adresses the fact that this generator is merely used as a proof of concept for the whole ltspice model, and that a more fitting generator should be used for DC generation from a residential VAWT. Whether such a generator with a lower field resistance would be physically possible to build all other parameters being equal needs further investigation.

There is also the potential issue of the order of magnitude of inductances specified in table 5 of (1), in the H range instead of mH, as inductances in the several thousand mH are usually found in generators in the thousand MVA power rating range

Determining empirically inductance parameters for the model to converge

Fortunately, there are some way to constraint inductance parameters in the range of physical realisability

Basically, the ratio of mutual inductance (stator to field) to the square root of the product of the stator and field self inductances. In the Ltspice model, rime varying self inductances of the stator are acessible through V(laa), V(lbb) etc… And self inductance of the field winding is Lffd plus the air gap self inductance.

This ratio can never be > 1, And in practice is low as the windings (field and stator) do not share a magnetic core, but are coupled through an air gap.

The following master thesis from 1976 tackles the issue of lower and upper bounds of physical realisability in terms of inductances. It is also one of the very few papers that provides numerical values for inductance parameters in the abc reference frame.

(2)

https://ttu-ir.tdl.org/bitstream/handle/2346/15506/31295015505711.pdf?sequence=1

The main issue, even when in a range of coupling conservatively low, is that the simulation speed is very sensitive to this ratio : if Lafd is raised from 0.19 above to around 0.69 or so (Laa0 and Lffd being constant) the simulation speed is reduced by a factor of around 2000. (Using the parameters of paper (1)) àThe mutual inductance between the field and stator windings is the main parameter influencing the magnitude of the induced Emf on the stator.

Also, Stator leakage inductance could not be derived from the paper, so it was derived independently. Note that bounding constraints arise from the fact that mutual inductance between windings cannot be higher than the square root of the product of the windings self inductances, assuming a coupling factor of unity. If an approximation of the coupling factor is done, based on fundamental air gap distance from the field shoe and stator slot, the stator self inductance upper limit can be derived. Moreover, a physically impossible parameter input gives rise to convergence issues or performance issues almost immediately in our model, Which helps in determining adequate values.

Of course, this methodology allows to model a somewhat physically realisable generator, not necesarily one available in the market.

As for the stator self inductance, it seems that it can be derived easily if zero sequence inductance is given in the datasheet, through the formula of L0 in figure 7, and Loavg in figure 1 of the present article.

Mechanical System Model

The mechanical system of a synchronous generator is based on the torque balance equation. It takes 3 inputs, namely electrical torque Te, mechanical torque Tm, and friction and windage of the generator rotor, Tfw.

Electrical torque Te is give by the formula input of the B12 source. This formula requires first the computation of the dq0 transform, also known as Park Transform, for flux and currents in the dq0 frame, as seen in Figure 7

Tm is derived from the VAWT model power output divided by the VAWT rotor rotational speed in radians.

Figure 7 dq0 transform
Figure 8. Electromechanical coupling. The V2 behavioural source is used for debugging.

Note that the V2 behavioural source is used for debugging, ramping to a constant mechanical power input. The actual source of VAWT mechanical power, source B21 is shown in figure 10.

The torque balance takes into account the torque ratio due to the gear box. generator shafts and VAWT shaft angular speeds are also proportional to the gear ratio.

generator shaft acceleration is given by the torque balance divided by the total shaft inertia as seen from the generator reference, that is, the contribution of the VAWT rotor assembly has to be multiplied by the gear ratio, and added to the generator rotor inertia.

Gear ratio is assumed to be < 1.

The model for the VAWT turbine is explained in detail in

(3)

https://www.econstor.eu/bitstream/10419/244151/1/1775681092.pdf

Figure 9 Mechanical assembly and VAWT parameters
Figure 10 VAWT mechanical model and wind input

Electrical load model

The electrical load is a 48V lead acid battery bank. The very crude model sets the battery electrochemical potential at 12.9V, under which no charging occurs. Battery internal resistance depends on battery capacity and state of charge (SoC). Charging current at a given voltage is initially slow for deeply discharged batteries because of high internal resistance of battery cells. It thens ramps up with decreasing internal resistance as SoC rises, to again decrease near end of charge, but not because of internal resistance this time, but because the electrochemical potential rises as battery charges. Due to very long time constants of battery charging processes, (several hours) at 0.1c/hour conservative charging strategy, it is unrealistic to simulate a whole charge cycle and thus a static model is sufficient.

The battery requires a DC charging circuit, ideally at constant current for the bulk charging phase. This is provided by a 6 diode 3 phase (passive) rectifier bridge, followed by a smoothing capacitor. This forms the output of the unregulated DC link.

For regulation, We decided to model a circuit that transposes well to a digital control strategy instead of an analog buck converter control IC mainly for two reasons :

Digital control allows full flexibility to implement a control system for the buck converter as well as field excitation control, wind sensor input, AC frequency input, fault detection input, battery parameters and monitoring input, etc.. It also allows experimentation to optimize the algorithm in order to achieve MPPT.

The second reason is simply to speed up the simulation, if proper care is taken to avoid convergence issues arising from some expressions like “IF” in behavioural sources, where in a lot of cases, it is better to replace them with differential Schmitt triggers.

A buck converter is preferred to a buck boost converter for this design to avoid drawing power at low rpm (when the VAWT power coefficient is low), which could lead to the VAWT turbine to stall. This logic is taken further to implement a threshold at which the switching starts from open circuit, at an input voltage well above the battery bank voltage. This forms the uvlo logic (under voltage lock out). Under that voltage (plus hysteresis), switching stops and the load is disconnected.

input overvoltage is a protection mechanism for the load, at which switching times are unreasonably short (low duty cycles) and exceed the operational enveloppe of the inductors, and Vds for the MOSFET. Of course, disconnecting the load at this point could lead to a runaway rpm overload of the turbine and generator and damage.

That is why wind turbine controllers have dump load terminals. in case of high wind speeds, or if the load cannot absorb more power because of low demand and or a fully charged battery, a crowbar circuit diverts power to an ohmic load, typically a large rheostat. This has the effect of lowering rpm and unregulated DC voltage at a given excitation level, thus protecting the turbine, diodes and MOSFETs. Whether the dump load performs useful work depends on the setup. for high geographic latitudes, high constant wind stations, the dump load can be used for heating an enclosed space to keep electronics at a safe temperature, it may also serve as a cold water network preheater, to avoid pipe ice damage. The use of non resistive loads like inductive loads, linking the generator 3 phases to, let’s say a 3 phase induction motor acting as a water pump is trickier, since care has to be taken that the crowbar activates at an electrical frequency within the motor frequency range. Frequent startups of an induction motor at high voltage and high frequency lead to a premature failure of a motor due to high starting currents. If that were not the case, star delta starters or VFDs would not be a thing. As for reactive coupling, synchronous generators provide reactive power, that can be consumed by induction motors. Moreover, the use of a passive crowbar triggered by high voltages may give rise to voltage waveform no sinusoidal in nature, even more so if the DC stage and regulator is not disconnected and keeps drawing power, as we will see later. It is recommended to devise a complemetary passive method to disconnect the DC stage when the crowbar operates, if one wishes to experiment with inductive loading of the crowbar circuit.

Grid forming or Grid supplemeting setups, using grid tie inverters are the most sensible way of performing useful usage of power, but they are outside the scope and intent of this article, that focuses on small setup islanded generation of DC power.

Let’s get back to the model now.

Figure 11 Load and Field electrical parameters

Here we specify the load (battery bank) internal resistance as well as the DC field excitation voltage.

AC/DC Conversion and Load Regulation, and battery charging

AC/DC conversion is straightforward.

As for the load, the model includes a basic buck converter used to charge a 48V battery bank. It does not take into account and additional DC bus to power a load besides the battery, also the control algorithm is just an example, gain coefficients are not optimized, and a state of the art charger would probably achieve MPPT based on a sliding mode control for better efficiency and to make the system state stay in its safe operational enveloppe. sliding mode control is part of modern control theory and an advanced design choice that is outside the scope of this article, and would be adequate given the complexity (large parameter space state) and non linearity.

Note that the control mechanism does not involve generator excitation control. This will be explored in a continuation article.

Figure 12 3 phase, 6 diode bridge rectifier
Figure 13. Basic ‘idealized’ DC buck converter for digital control strategy

This is an idealized proof of concept version since it does not make use of a gate driver using instead a VCVS (E1) as well as for current sensing (E2) and also uses ideal diodes, as well as a crude battery model. The rest of the control algorithm is meant to represent digital control. The use of a first order LP filter for in_fb eases convergence since the signal is noisy, Hysteresis and rise/fall times of the schmitt triggers also help.

Let’s cycle through the main parts of the controller.

V6 and B18 signals are fed to the differential schmitt trigger.

The schmitt trigger compares the signal that represents the duty cycle to a sine wave of switching frequency, with DC offset equal to amplitude, the result is a varying duty cycle square wave signal at switching frequency.

This signal is sent to a VCVS (E1) that is meant to represent a gate driver, like the IR2110S IC. Its gain is 3, so as to drive the high power mosfet to a sufficiently high Vgs voltage, at around 20V.

The duty (base) cycle is calculated with the standard buck converter duty cycle formula. instead of taking the maximum expected unregulated input voltage seen in most application notes, it takes the present unregulated voltage. unreg_dc(). All of this is multiplied by the efficiency factor of the buck converter.

In essence, this forms an open control loop which is based on theoretical values and if subjected to calibration would give better output voltage regulation.

Since this is not enough for real world scenarios and calibration is not always possible independently for each device, a closed control loop helps in regulating the output voltage. The open control loop formula only helps to give a setpoint duty cycle from which the controller should start switching.

The closed control loop negative feedback is given by the 2*(4 – (V(feedback) – V(vss)) term in the duty() function.

This ensures that the output voltage discrepancy from the open control loop is corrected. Note that there is no compensation network filter in the feedback loop. Those can be implemented in analog form or digital form.

Also note that duty_base() is 0 if the controller detects an uvlo condition. this protects the VAWT turbine from stalling by disconnecting the battery load.

dutybnd() is just a numerical conditionner to prevent duty cycles > 1

The boost() function ensures that the voltage, and hence the current flowing into the battery rises after the unregulated dc setpoint defined in the boost() function is crossed.

At this point, the feedback from the output will be dictated by the current control loop formed by E2 and D9 once the current threshold is crossed, and will oppose the boost function. The DC equilibrium point is defined by the crossover point of the boost linear function and the output feedback function, which mainly depend on their respective gains. It is preferable for the output feedback function to takeover the boost function early after the set charging cureent threshold to keep battery charging current, inductor current and MOSFET average and peak currents within their nominal ranges.

An additional protection layer is provided by the ovlo() function that kills the boost function. Once the ovlo threshold is crossed, the dump load crowbar should be activated to protect the MOSFET from high Vds.

Simulation performance considerations

Care has been provided in the DC load model to ease convergence. Simulation speed is inversely proportional to the rotor assembly rotational speed. Also, an important factor that slows the simulation is the switching frequency of the buck converter. Given the long time constants required to produce meaningful data, it has been kept at a really low value of f_sw=500 Hz, compared to usual converter designs.

Annex A: Generator fault testing.

The following circuits were used to test the behaviour of the generator under load rejection and 3 phase short conditions to check for adequate response.

Figure 14. 3 phase Short Fault circuit
Figure 15. load rejection test circuit

Annex B : Overvoltage and mechanical overload protection of the VAWT

High wind conditions and unability of the load to absorb power because of charge termination or low power demand downstream of the battery on the DC bus may cause mechanical overload of the VAWT, too high generator rotor speeds, too high fluxes, heating, arcing, and overvoltages that can exceed winding insulation dielectric strength and cause the winding to fail.

Some high end HAWT can be feathered by adjusting the pitch angle of the blades to decrease wind coupling. Variable pitch VAWTs or adjustable vanes can be designed to protect the turbine at the root level of the issue, but these complexify the turbine design.

The low cost method involves electrical braking by dumping excessive power into a dump load, through a crowbar mechanism, This will decrease RPM, but the whole assembly will still be subjected to high torque conditions.

As already mentioned, the crowbar can be implemented on the DC bus or on the AC bus. Since this is a critical safety mechanism, care is needed to make it work in a failsafe and passive manner, without any high level input from an IC or microcontroller, or at least have a completely passive system to supplement the active one.

A passive system on the AC bus can detect overvoltage that signals high wind conditions or underload, and be implemented through TRIACs, one for each phase, and triggered by a current pulse through the gate that is initiated when series back to back Zeners or a single TVS diode start conducting, Once the line to gate voltage is above their breakdown voltage. A current limiting resistor should be put in place to limit the current flowing to the gate below Igtm. Note that it is more a continuous trigger that persists while the overvoltage is present Note that the trigger ceases when the AC waveform goes below the TVS or Zener voltage. In that condition, the TRIAC still keeps conducting until current gets close to 0 in the AC current flow. After that zero cross, the TRIAC won’t conduct until the voltage threshold of the TVS/Zeners is crossed again.

<Check> The resulting AC characteristics seen by the dump load are not sinusoidal but chopped, and should not be used to drive an inductive load like a motor. The strategy mentioned before to use an induction motor as a dump load that perform useful work is possible through active switching by a contactor or relay for instance, if the motor is operating within voltage limits and volt per Hz limits that would arise in worst case turbine overload conditions at the time of startup. That generally means the the motor should be over-rated in terms of power, so it would operate well below nominal conditions, taking into account fluctuations arising from the unpredictable wind conditions that impart stress on the motor, and repeated start/stop cycles. An inrush protection device could be envisioned, and contactor hysteresis should also be taken into account to limit start/stop cycles. The TRIACs overvoltage protection would still be used at a higher overvoltage trigger level as a last resort protection and power three rheostats.

despite its higher complexity, the AC dump load strategy also has the advantage of diverting the current before the rectifier bridge, and thus limit current and heating stress on the rectifying diodes.

DC crowbar protection.

This design is easier to implement, requiring a single SCR and a zener or TVS triggering mechanism, before the switching mosfet stage. it provides unregulated DC (with a substantial amount of ripple) to a load, ideally a rheostat. Since the crowbar operates on DC, there is no current zero crossing, and it will stay in forward conducting mode as long as the generator and VAWT keeps turning and provided that the buck converter shuts itself off (zero duty cycle) to prevent the battery back feeding into the dump load.

Reverting to open crowbar can be done by shunting the anode and cathode of the SCR temporarily through a previously charged capacitor, so as the anode sees ground potential and the cathode sees Vdc. (essentially the SCR sees a reverse voltage pulse) The reverse discharging of the capacitor is accomplished through a MOSFET or relay. The capacitance of the capacitor must be sufficiently large to provide a discharging time constant longer than the Toff parameter of the SCR and also depends on the dump load impedance

Note that in this setup, the SCR is a low side switching device, the cathode of the SCR being at ground potential. See figure 16.

Dump load considerations.

We will focus on a resistive dump load as this kind of load offers the maximum flexibility and ease of design and safety of operation, as well as optimally controllable for adequate braking and overvoltage protection.

We will explore two control methods to perform adequate control of the dump load operation.

One based on electromechanical impedance matching : Instead of using a switching device to perform impedance matching, a servo actuated rheostat would be used. A rotary rheostat is prefered to obtain fast response using a stepper motor, However, for testing purposes, it seems that linear rheostat of high power > 500W are cheaper and more easily available. That would require the use of a linear actuator that has comparatively slower response times.

Note however that high dR/dt result in high torque fluctuations.

Also, A control law based on crowbar electromechanical impedance matching operation between a low and high setpoint of a fast reaction variable like DC voltage will introduce generator hunting effects and the mentionned high dR/dt, essentially the control mechanism resets the impedance at a high value once crowbar is deactivated (no current flowing through the SCR). This introduces oscillatory behaviour, which gives rise to constant high amplitude motion of the servo actuator, introducing wiper fatigue and high torque fluctuations, and a suboptimal braking. A control based on a slower variable like shaft speed is thus preffered.

Care has to be take for adequate thermal management of the rheostat, since it could operate on a substantially low fraction of the total winding number, and create hotspots. A forced cooling method or operation of the rheostat in a thermally effective medium such as transformer oil may be necessary, and would significantly increase device complexity due to safety requirements. This would involve the design of the dump load the same way as a large transformer is built. Produced heat could be used offsite.

Excitation control during crowbar operation, static impedance dump load.

The other method assumes a static load impedance and varies excitation level upon crowbar activation according to a control mechanism specific to crowbar on operation mode. In this case, part of the Joule heating is dissipated in the generator due to the substantial rise of excitation current to achieve adequate braking.

Direct Heat transfer through magnetic braking.

This method would add an intermediary rotor between the VAWT and the generator with a permanent magnet arrangement, Magnets should exhibit high Curie temperature, according to projected maximum temperature rise in the brake rotor through radiative and convective effects. A claw like copper heat sink would be engaged radially to modulate the braking effect arising from Eddy current induction. The copper heat sink would be fitted with copper heat pipe L shaped protrusions, that would sit partially in an effective thermal medium stored in a tank. The issue with this approach is that engagement of the claw is a mechanical process that would involve linear horizontal heat pipe motion, which would give rise to a challenging task of making the thermal medium tank airtight, let alone pressurised. Another issue are axial and radial force components on the claw mechanism, which would require adequate sturdiness of the claw engagement/disengagement actuator. Economic factors should also be taken into account, as permanent magnets are costly due to the rarity of the source materials.

Prevention of VAWT stall.

Prevention of VAWT stall is usually done by properly selecting a recovery voltage under which the dump load SCR should stop conducting. SCRs only stop conducting if the current flowing through them reaches a determined threshold provided in the datasheet, Which is usually quite low. One way should then be to provide a very low impedance path to reduce current flowing through the SCR (shorting the anode and cathode by high AWG gauge wire and a sufficiently rated relay in terms of current. One alternate way is to pulse-reverse bias the SCR to force it into reverse-blocking mode. This can be done by charging an AC capacitor and shunting the positive lead to ground (vss) when we want to block the SCR, This is done by a relay or MOSFET. the other lead of the capacitor is connected to the SCR anode, creating a short term negative voltage with respect to vss to appear at the anode, effectively bringing the SCR into reverse blocking mode.

This would need a high voltage, high capacitance AC capacitor, commonly used on electrical motors. These are bulky and quite expensive. In our simulated crowbar, a 450V AC rated, 20µF capacitor is used.

The circuit used in the crowbar simulation assumes that the battery charging bus is disconnected, that is, the battery does no accept more charge. In reality the PWM or MPPT block used to charge the battery would be still connected, it is no simulated here mainly for performance reasons.

This means that the rotor has no load and is essentially freewhelling until the crowbar activates.

Wind speed is ramping up and reaches steady state at 1 sec into the simulation.

The crowbar activates when the DC voltage reaches 250V and resets at around 48V DC.

There are several cycles of activation / deactivation until the VAWT net power is sufficiently high so that the voltage does not fall under 48V DC. At this point, the crowbar is permanently on.

The crowbar impedance and power rating has to be selected carefully so as the VAWT turbine remains in the rotor speed and torque operating enveloppe up to maximum rated wind speed.

Figure 16 : DC crowbar circuit

230VAC to 48V, 1400W Lead-Acid battery bank charging circuit

Model download at the end of the article

This is the complementary circuit to the 48V to 400V converter, doing the opposite conversion

However, it is presented here as a simple charger directly tied to mains without PFC.
The input line filter has been omitted for simplicity.

Since the charger assumes the presence of an AC link (230V AC for that design), logic power is supplied by two small AC/DC 50 Hz 3W transformers.

They are not modeled precisely to the specifications in this design.

The first transformer powers one LM317 regulator to provide bias voltage of 10.5V to the switching LTC3721 IC and to the optocoupler transistor. While the second transformer powers one LM317 set at 5V regulated output for the MCU, the LT1006 op-amp, and the AD8418 current metering op-amp, as well as the LM113 voltage reference diode used by the AD8418; And another LM317 with an output voltage set at 12V. The LT1001 used in the Howland current pump requires a 12V supply in order to source an adequate current level.

The circuit has been simulated up to 25A charging current for a 48V SLA battery bank. (Assuming an individual battery voltage of 12.4V)

Due to the simplest model used for the battery bank, the actual behavior to reach a steady state may be different.
CC is achieved by high-side current metering, whose signal is compared to a bias level output from a DAC.
The higher the DAC output level, the higher the charging current.

High-side current metering is preferred for chargers since it can detect load shorts and offers better noise immunity.
Here we use a dedicated AD8418 IC for that purpose.

This approach is failsafe in the event of a MCU failure since a 0V DAC voltage output would command a 0A charging current.
CV for float/trickle charging is achieved by varying the wiper position of a 5K I2C digital potentiometer, controlled by the MCU.

Note that the circuit has been tested on an ohmic load (10 and 25 ohm) for stability.

It could also be used as a versatile CC/CV PSU besides charging

Using the circuit as a charger for a 24V battery bank could be envisioned, but has not been tested for performance and stability at the time of publication of this article

As for the rest of the circuit, it is more or less the same as the 48V to 400V converter from the preceding post.
Due to higher output currents on the secondary, choke, the output power level has been derated.

As for stability, there is no perceivable ripple in the output up to 25A.

Assuming a charging current of 0.1*C (C being the battery capacity in Ah), This charger could theoretically charge a battery bank of four 250Ah batteries at a nominal rate.

This circuit is the simplest expression of a CC/CV charger. It does not perform a battery bank voltage check before charging, temperature compensation, or coulomb counting. These features require further MCU / digital side control and are not expected to be modeled properly in Ltspice.

Speaking of digital control, It is expected for the MCU to monitor the charging current through the AD8418 so as to set the DAC voltage “curr_offset” to perform the appropriate charging program, as well as to monitor the bank voltage, as a digital control outer loop.

Efficiencies (simulated)

Pout/Pin. Pin taken at the node after the full bridge rectifier.

5A0.951529
10A0.945238
15A0.942658
20A0.93861
25A0.933888
30A0.930961
CC mode (stepping curr_set DAC voltage)
CV mode, stepping 5k CV digipot. 10 ohm load.

Ltspice Model Download :

Ltspice Model of an isolated, 48V to 400V, 1600W, Push-Pull converter using Chan model inductors, using the LTC3721 IC.

And detailed design information

Disclaimer:

The following design comes with no guarantees whatsoever. Although a decent amount of time has been spent to ensure that the model works well over the whole range of its design constraints, some errors may still linger, Some more experienced engineers may find some design choices questionable, If you’re able to optimize it or build something better around it, that’s nice.

The choke inductor and the transformer may be a little over-engineered and drive the costs up, given the large ferrite core choices.

You may be able to extract more power than 1600W, but tread carefully.

Additional Resources :

The download is available at the end of this article.

There is also the ‘sister project’ to this one, which performs 230V AC to 48V DC conversion, with the intent of battery charging : It is designed to allow current/voltage control via DAC and a digital potentiometer, so you can digitally control voltage and current and design a charging program.

It is available here :

Abstract:

The main goal of this model is to serve as an aid in learning about the Push pull converter topology,
it also could prove useful when building a prototype, as a part of a UPS or solar converter design.

Care has been taken not to overburden the simulator and allow reasonable simulation speeds.

Most of the simulation time is spent ramping up the voltage (soft start)
Using smaller starting loads and stepping them once the converter is fully started decreases simulation time, Once you’ve determined that the soft start ramp is ok.

Intended Audience:

Makers or junior power engineers with little experience, looking for a project that can lead to prototype build.

Three models are supplied :

  • The fastest one uses linear inductors, voltage sources for IC power, no isolation, and a simple feedback circuit without optocoupler isolation.
  • The second one is the same as the model above but with non-linear inductors (Chan model)
  • The full model uses proper and more realistic component DC supply schemes as well as non-linear inductors, as well as isolation. Note that Ltspice is not always able to process isolated secondary circuits, even with the use of a stitching resistor, unless it is very low resistance. Here the problem appeared so we linked both grounds. A practical circuit, of course, will not have that constraint.

Design Parameters

  • Max continuous power, 1600W max. Inductor Thermal effects are not modeled, derating may be advisable, Although a large saturation margin has been taken into account.
  • Input from 48V (discharged battery, UVLO threshold) up to 57.6 (bulk charging voltage when a charger is connected)
  • 400V DC output. it supplies the same voltage as a PFC would.
  • This allows easier load switching or load sharing between the battery source and the AC / PFC source converter and can be adapted to larger designs.
  • Optimized for high power.
  • Moderate to good efficiency 0.93
  • Low cost. (uses powder core inductors)
  • Fully isolated design.
  • Under Voltage lock-out set at 48V to prevent battery bank over-discharge
  • 4mm² wire for primary of transformer, 2*24 turns, center tapped
  • 1.5m² wire for secondary of transformer 2*234 turns, center tapped
  • T400-26 transformer core (Iron powder)

As said before, it is for teaching or training rather than commercial purposes.
It will require hand winding of the toroid to build a prototype. (which is a cumbersome and long process, It is advised to watch several videos about that art to do it properly the first time. (You do not want to rewind it a second time). Building a toroidal transformer is a valuable learning experience.

Some Toroid Winding tips :

Use proper dielectric insulation between windings to control parasitics, it also serves to protect the windings from abrasion.

  • Use a counter to keep track of turns.
  • Keep the winding tensioned so it does not spring back.
    You can also ask toroidal transformer shops to build you a custom transformer according to your specifications.
    Proper care has to be taken to balance the primary windings around the center tap as equally as possible to avoid flux imbalance. Fortunately, the number of turns of the primary is low.
  • Strategically place the tap halfway through the core height, and wind the primary legs as symmetrically as possible, by adequately controlling the winding pitch. Looking at resources explaining the proper way of transformer tapping is advised.
  • To make enamel wire solderable, I usually use abrasive dremel cylindrical tips. Do not use a flame as the enamel is very flame resistant and that will anneal the copper making it more fragile.
  • You can also watch videos from HAM radio makers, as a lot of them have mastered the art of core winding. A balun is not exactly the same as a push-pull transformer, but a lot of practical building tips apply.
  • Always check the final product inductance while testing on exposed wires, with a margin of excess wires so that you can always add turns if inductance falls short. Fortunately, transformer inductance is not that critical, what is critical is balancing and the turn ratio.

Note that It will be hard to find an exact turn ratio from commercial solution catalogs, but you can always look for and adapt the design accordingly. A different ratio will affect the minimum and maximum duty cycles of the converter which could make achieving the 400V target harder at the nominal output load of 4A.

Capacitor Considerations

Input capacitors do not need to be large because of the low battery impedance and stable voltage.

Output capacitors should be low ESR, We choose expensive high capacity, high-voltage electrolytic capacitors to maximize hold time and for lower ripple, and to make the choke
sizing reasonable.

Hold time considerations mostly depend on the load parameters
Additional 450V MPP film capacitors were used. Do not use X2 line capacitors as they are designed to fail short.

MOSFET considerations.

In our design, each MOSFET is subjected to 25 A average current with peaks of up to 120 A due to parasitics (at turn-on), with low-value gate stoppers resistors.

Infineon’s IPP110N20N3 are rated at Id of 88 A and pulses up to 352A.
The datasheet is available at :

https://www.infineon.com/cms/en/product/power/mosfet/n-channel/ipp110n20n3-g/

Thermal management of MOSFET is of utmost importance a clever prototyping solution could make use of radiator/fan bundles designed for CPU cooling, as they often integrate heat pipes, This would complicate the layout significantly and the total prototype volume because of the sheer bulkiness of these kind of components, and require drilling the heatsink backplate. A single modern and standard CPU heatsink fan can cool loads of around 100W. Calculate maximum MOSFET losses and design the solution accordingly.

Never operate MOSFETs under load without proper thermal management!

Overcurrent Protection.

The Switching IC provides a hard current limit that stops switching in case of overcurrent and a resume algorithm explained in the datasheet. For soft current limiting and output CC, you’ll have to implement a current monitor yourself and a feedback signal that overrides the CV signal fed to the optocoupler in case of an overcurrent situation that would decrease output voltage.

Transformer considerations: turn ratio, switching frequency, minimum inductance.

Check this resource for basic formulas.

https://www.analog.com/en/technical-articles/high-frequency-push-pull-dc-dc-converter.html
Also, check :
http://tahmidmc.blogspot.com/2013/03/output-inductance-calculation-for-smps.html

Inductance range tolerance is high since the push-pull converter is based on transformer action. It is not a critical design parameter.

The allowable duty cycle range will dictate the maximum voltage differential between the primary and secondary, in combination with the output voltage range. The fact that the input could already be thought of as regulated (it is a battery, but may be subjected to higher voltages seen by the converter during charging) and that the output voltage is designed to be kept constant eases the design.
It is however important not to drive the core into saturation, So we have made the choice of a large T400-26 iron powder core, although thermal effects (and the decrease of inductance caused by it, will play the limiting role rather than saturation. Here the margin is <add figure>
A larger core also allows a lower fill factor which will improve cooling and reduce ohmic heating from the current flowing into the conductors.
It also makes manual construction easier.
As said before Controlling the winding balance in the primary is critical in Push Pull converters.
An imbalance gives rise to a Bias flux buildup that decreases efficiency.

A low fill factor also allows better cooling performance, even more so if forced (fan) cooling is used to blow axially through the core.
When building a transformer, optimization is complex because of the large parameter domain.

For such a design, it is advised to perform a faster simulation using a standard LTspice coupled inductor,, and specify coupled linear inductors inductance based on Push Pull converter design formulas, and check that the circuit performs ok on that basis. taking care efficiency changes into account with each parameter changes, and performing load stepping, and voltage input range stepping. Efficiency should be high using a fully linear transformer.
If not, something is amiss. Remember that efficiency also depends on load, and is usually lower at very light loads.

On the basis of this first design iteration look at the current flowing through the primary (RMS and peak values), and knowing the number of turns you’ll get the H field strength, which will be used to get the B field strength. B = H/(µo*µr) Then you can look at material tables and check that you are operating within a safe margin. It is a bit complex because there are derating factors, for instance, because of frequency.

Thermal runaway is the situation you want to absolutely avoid (It decreases inductance, which increases the magnetizing current, making the core saturate even more and induces losses which translate into thermal runaway)
Our strategy is low-cost-driven. We choose low permeability iron core and decreased switching frequency to minimize core losses. (iron powder cores perform better at lower switching frequencies)
Iron powder cores are low permeability because of the distributed air gap between magnetic particles, and exhibit high (around 1.5 T) and soft saturation.
For the main transformer, we choose the low-cost 26 Material, and selected a large T400-26 core, allowing higher fill factors. The transformer turn ratio requires a large turn number for the
secondary.
For the output choke, we used a lower permeability 18 Material and a quite large stacked core. Output inductors/chokes operate at a high current bias level so that it derates inductance. Inductance value is also a critical design parameter. When using a suboptimal (too low inductance, output ripple, and inductor heating will be higher, with also a risk of thermal runaway and gradually decreasing stability.) We noted that when using too low inductances, the design refused to reach our target voltage, Which seems to be a protection feature of the converter IC.

To sum up, We have to obtain an inductance large enough for our filtering goals at nominal power while
reducing the bias B field. B field is proportional to the number of turns times permeability, all other parameters being equal, while inductance increases with the square of the number of turns times permeability. Thus it follows to use a larger core with lower permeability to accommodate a larger number of turns to meet inductance requirements while staying under saturation levels. To lower the B field, we also used the stacking strategy to increase the total compound core area. It is easier to wind that way than individually winding inductors and placing them in series, on the board, especially if PCB real estate is a concern. It would however decrease cooling efficiency. Not seen much about that method in commercial products, as it could also increase flux leakage. Better test it.

EMI concerns.

The low-frequency operation of 25Khz reduces EMI concerns depending on regulations on that VLF band and the other components that may be subjected to it. It is above the audio hearing range, but some harmonics may find their way into audio equipment (The IC switching frequency may go up to 1Mhz, changing the frequency would require choosing a better-suited, ferrite instead of iron powder material and adapting core dimensions, usually smaller ones. However, we had trouble making the simulation run smoothly at higher switching frequencies. Higher switching frequency also has a dramatic impact on simulation performance, as the minimum timestep has to be lower.

Of course, general layout guidelines apply, such as reducing the loop area of switching components traces paths (MOSFET drain to source) and the length of gate signals. Shielding is an option if it does not interfere with cooling.

Lower frequency however could make the core produce an audible stridulation effect, because of the magnetostrictive effect that is close to the audible range.

Core design helpers :

Our advice is to use a Ltspice core test bench using the resources here :
https://www.eevblog.com/forum/projects/arbitrary-%28saturable%29-coupled-inductors-in-ltspice/

For more information about the Chan model :

https://ltwiki.org/index.php?title=The_Chan_model

You will also need these very useful resources :

  • Magnetics catalogs and materials datasheets from major Western manufacturers: TDK Epcos, Ferroxcube, Magnetics.inc Micrometals,
  • The same for Asian ones: JUNCAN, Tangda, Caracol, etc… A popular seller is Tangda if you need to source (relatively) cheap cores from China.
  • The B/H curves when you can find them, if the Chan model parameters are not specified.
  • Magnetics cross-reference lists, images and pdfs. This will make selecting cores a littles easier, when switching from one manufacturer to another.

If you need to look at a B/H curve or if you have experimental data from a B/H curve tester (usually pluggable to an oscilloscope), you will need to find the crossing points on :

the B axis (y) : for Br (B field – remnant) and Bs (B field – saturation)

and the H axis (x): for Hc (H field – coercivity).

Bs is the saturation (horizontal asymptote line B value) for hard saturation, for soft saturation it is determined differently, as the B field keeps increasing, albeit with a lower and lower slope.

A good rule of thumb for soft materials is to stay in the linear region, with a good (30 to 40% margin)

Fortunately, Chinese manufacturers provide the B/H curves and Br, Bs, and Hc.

With all these collected data, you are ready to test the cores in the Ltspice Chan model test Bench.

  • Use the manufacturer-supplied geometric data OD, ID, and Height. We updated the test bench with geometric data calculators for the magnetic length and area required by the model.
  • Input magnetic data for the Chan model, taking care to use SI units: Hc (A/M), Br (T), Bs (T), (Amperes/m and Tesla)

Alas, Western Manufacturers usually provide the Al value and Bs (B sat) but almost never an exploitable full BH curve. You will need to contact them for this, but it may be a trade secret, who knows?

What you can do however is parse scientific publications to get harder to find values such as Hc and Br, and the problem is that they come for generic alloys (says MnZn, or very exotic ones) A comprehensive database of core parameters is clearly needed at this point.

If the data is in Oersted and Gauss, multiply the Oersted value by 79 to get A/m and for Gauss divide by 10000 to get Tesla units.

Some reference data, mostly for iron core materials collected that you may find useful :

An important note that I have not confirmed at this point: Note that the Iron powder materials data found on Chinese resellers are (presumably) for a pure (no distributed gap) material, thus if you plug the data into the mag_inc_bias.asc, you’ll find an abnormally large Al value.

So the strategy is to set primary_turn in the test bench model at 1, and play around with the core Chan model parameter Lg (gap length value in meter), until you obtain the nH/t^2 that is specific to the core.

Remember to set I_bias to 0. There is also a 60nH inductor in series, that would need to be set to 0 for adequate measurements of very small inductances.. I have no idea what is the purpose of this.

SUMMARY OF INDUCTOR TEST BENCH SETUP

  • Create one asc file based on mag_inc_bias for each core (makes life easier)
  • Fill in geometric data
  • Fill in material data (Hc, Br, Bs)
  • Set primary_turn to 1.
  • Set I_bias to 0.
  • Find distributed air gap equivalent gap length L_g by trial and error (examine inductance .meas in the error log) until the inductance value is equal to datasheet Al.

This is particularly useful to test for inductance decrease due to I bias current (seen in the output inductor choke)
If you need to lower the testing frequency, you’ll need to increase the simulation time because the measurements use 15th/16th cycles of RISE/FALL for inductance measurements, otherwise, you’ll get “measurement failed”

Inductance measurements are required for the choke, for the transformer, just check that the B field remains under Bsat with some margin.

Note that the I1 source is used for inductance measurements (It is set at 1mA, thus the x1000 factor in inductance measurement)
The measurement is based on the formula V = Ldi/dt, L = V(dt/di) = V/freq2pi at zero cross.

Increasing I2 will decrease inductance This is used for the choke measurement (under DC bias). Test with I2 value equal to the max allowable output current, with the frequency set correctly and the choke turn number set correctly.
Verify that the inductance value is still above requirements and that the B field in Tesla is not above Bsat.

As a final note, It should be said that the Chan model has been superseded by the Jiles-Atherton model which shows better fidelity to the experimental BH curves.
Unfortunately, Ltspice models using the JA model (CoreJA) are prohibitively slow for use in power product simulations, But the test bench could be adapted using CoreJA. The advantage of the Jiles Atherton model is that you can find a database of JA parameters for a lot of cores in the magnetic.txt file of the ZZZ library. This is the famous Bordodynov library (also known as the Yahoo Ltspice group library or the Ltspice groups.io library) It is a must for every serious Ltspice user.

Software also exist that help in complete solution design with an emphasis on magnetics, such as ‘ExcellentIT’, and also good product finders on manufacturer’s websites, to help in core selection.

Once you have made a provisional choice for the cores, turn number and turn ratio, you can replace the linear models using inductances with the Chan model using the turn number.
The Chan model slows the simulation speed only very moderately.

Isolation

This is an isolated design, However, Ltspice complains when using separate grounds, unless stitched by very low resistors. here we used a 0 ohm resistor between GND (primary ground)
and COM (secondary ground) in practice of course there is galvanic isolation.

Optocoupler tuning

We used a TL431 to provide a stable 5V reference required by the optocoupler output transistor.
To provide current to the optocoupler diode we use a modified (improved) Howland current pump.

Using a simple shunt resistor of around 240k to control the current flowing into the diode induces noise in the simulation. It should be tested in practice. The advantage is that such a solution would be passive and not require a low-voltage DC supply operating on the isolated side.

More information about the improved Howland current pump is available here :
https://www.ti.com/lit/an/sboa437a/sboa437a.pdf

Compensation network

Compensation network time domain testing :

Replace the passive resistor load with an active load (flagged as load)
The compensation network can be tested for stability by stepping the active load, and examination of the induced voltage oscillatory response, its amplitude, and its damping characteristics.

For more information on compensation networks :
https://www.analog.com/media/en/analog-dialogue/volume-57/number-2/step-by-step-process-to-calculate-a-dc-to-dc-compensation-network.pdf

Ltspice (the latest version) also offers transient frequency response analysis. It combines transient analysis (so that the circuit operates normally), while a small signal
stimulus is provided on the input voltage side. The small signal response on the output is analyzed so that a Bode Plot can be drawn and analyzed for stability. (checking gain margin and phase margin taking into account the switching frequency vs the frequency location of poles and zeros.)

Combining frequency analysis with a transient analysis has the advantage of not requiring specialized frequency response models for IC (When they are available)
In this model, the input voltage is stable, Output capacitance is large with a low ESR, which helps for stability. A good test would be to introduce a disturbance by simulating a charging operation in the bulk (constant current) charging phase.

Powering up the IC, the Optocoupler, and the current source OpAmp

The switching IC has access to the primary side power battery power. As it is a well-behaved supply, no need to power the IC from an auxiliary winding from the main transformer. Powering the IC is documented well on the IC datasheet. In our case, however, the design is simpler.

Note however the presence of the R33 resistors that shunts some current from the primary DC link into the IC, charging the capacitor faster, than what the LM317 alone would do, and allowing the IC to start faster, The datasheet uses a 2k value for ar 12V primary, we just scaled it linearly.
In this design, we used a simple LM317 regulator, which may be used to power other logic loads. The LM317HV version tolerates the battery bank voltage. You can also use a lower voltage version and power it from a single battery unit closest to ground, Which would have its positive terminal sitting at 12V above primary ground. Note that the IC is internally regulated at 10.5V or so, and can operate as per datasheet with as little as 8V. We found that it needs 10.5V during startup, and we set up the LM317 to supply a constant 10.5V. The absolute maximum rating is 12V. We also used a pre-regulator high voltage 60V Zener to protect the LM317 in case of a voltage transient. (Which could come from the charging operations)

For the secondary, things are a bit more complicated. The only active component here is the OpAmp of the Howland current pump driving the LED of the optocoupler.
In reality, it is almost guaranteed that other logic or control components will be operated at low voltage with a secondary ground reference. Thus we used a 5V setting for the secondary side LM317, This low voltage did not seem to affect negatively the operation of the Howland current pump opamp

We could assume that the 400V link always has access to power, for instance, a rectified mains AC power source output from a 400V PFC unit.
In UPS and solar applications, that may not be always the case, take as an example the “cold start” of an UPS from the battery in the absence of mains power.

The absence of power to this component means no voltage feedback signal to the switching IC. It needs to power up quite fast (well before the secondary reaches 400V DC)
For this, we use a secondary auxiliary winding, a rectifying diode, and an LM317 set up for 5V output to power the OpAmp. The LT1001 Op-amp. is fully turned on at around 2.5V

An optional Zener could be added as a TVS function to clip transients above the LM317 rating.

MOSFET parasitics, ringing, and leading-edge current spikes.

Figure 4 shows leading edge current spikes, they are not associated with ringing (as they are fully damped). The following thread identifies the culprit as being the reverse recovery time of the secondary side rectifying diode as well as the gate pulse. To minimize these effects, One can use fast recovery diodes for the secondary rectifier, as well as to increase the gate stopper resistor values (but the latter has drawbacks, as we’ll see in a moment). Reducing these spikes by using fast recovery diodes may increase overall efficiency, as well as decrease HF EMI (the spike frequency is substantially higher than the switching frequency).

In our simulation, the current spikes are well under the 352A max pulse current specification of the MOSFET, so it should not damage the MOSFET over the long term. (When using standard silicon diodes for the secondary’s rectifying diodes.

https://e2e.ti.com/support/power-management-group/power-management/f/power-management-forum/680959/lm3481-lm3481-current-spike-when-mosfet-turn-on

Voltage transients and ringing.

Although this model does not exhibit this unwanted phenomenon, it is probable that a real-life implementation would because of parasitics that are not modeled here.

Ringing comes from MOSFET parasitic capacitance, coupled with the driven circuit (a transformer) inductance as well as trace inductance. Most of the push-pull converter designs come with some sort of snubber (RC series circuit across drain and source, tuned to the problematic ringing frequency), However, Value tuning is quite layout dependent. Having broad but short gate traces also helps in the management of the problem. make sure your layout accommodates some room to add a snubber.

A resistor gate stopper (here 1 ohm) also may help, but its value cannot be pushed too high: You also have to take into account gate capacitance. A large gate capacitance cannot suffer from a too-large gate stopper resistor, or the MOSFET will turn on slowly, and the slow turn-on will increase average Rds. On the other hand, a too-low gate stopper could push the gate currents above the IC specifications, especially if the MOSFET gate capacitance is high. (Which is somewhat the case for high-power MOSFETs) Thus a snubber seems like a good option.

Remember that this part may exhibit different behavior in real life due to the non-modeling of all parasitic effects and their layout dependence.

Using a 4-ohm resistor instead of 1 ohm decreases the peak pulse current from 210 A to around 160 A.

Figures

Figure 1: Voltage ramp-up / soft start. Load stepping after steady state is reached
Figure 2: Voltage transient / Stepping load from 0A to 4A
Figure 3: Voltage transient / Stepping load from 4A to 0A
Figure 4: MOSFET Drain current (spikes due to gate pulse and secondary diode recovery)

Model Information

LM317 model as well LTC3721 should be present in a recent Ltspice installation.

You may need Infineon’s IPP110N20N3 model.

This model has been tested on a Ltspice installation using the ZZZ (Ltspice groups.io community library), it is advised you install it.

Download

SC-120W 12V PSU with UPS function review.

or How to battery backup your router the proper way.

This unit is a good choice to provide long UPS time in an event of a power outage for small but critical loads like an Internet router, Where a full 110V/220V UPS is overkill. A typical internet router power rating is around 15W. Assuming the SC120W is wired to a 60Ah 12V battery and assuming 50% discharge, the router could keep working for almost 20 hours in an outage event.

Voltage Setup.

There is a white screw near the battery cable. This regulates two outputs:

  • The set float voltage of the battery (for a lead acid battery, this should be always set up at at least 12.9V +/- some voltage according to the ambient temperature, but it will be always more in a realistic scenario, the float voltage is between 13.1V and 13.8V. That means that this unit is not made for frequent outage cycles, since it would also cycle charge the battery, and charging is usually done at higher voltages.
  • This screw also regulates the working voltage of the load, minus a diode drop it seems (that is, the load voltage will be the battery float voltage minus 0.6V)

So, for example if you set the battery float at 13.8V, the load will be supplied 13.2V. Routers have voltage regulators inside. As a rule of thumb, electronics tolerate +/- 10% deviation of the nominal supply voltage which is 12V for most routers, Putting 13.2V into the router could damage its regulator over the long term, leading to reduced lifespan.

The proper way to do it : Insert a DC/DC step down module like these based on the LM2596A between the SC120W and the router,

DC/DC Step Down to convert 13.8V into 12.V. The 3A rating is adequate for most routers

and adjust the step down module to output 12V, now you can set up an appropriate float voltage for the battery without damaging the router.

Charge current limiting.

There are no regulating screws for charge current limiting, so, if you set the voltage too high and the battery is discharged, it could draw current up to the overcurrent protection limit. I did not push the test unit far enough to see if it is 10A or less at (the PSU nominal power rating), Nevertheless, it is recommended to charge a battery at a current no more than 0.1 * total capacity. If you use a small battery, and there are some outages, reduce the float voltage accordingly. Best is to use a large battery so you can setup a higher float voltage, its lifetime won’t be affected by the charging currents which would fall mostly under 0.1 * total capacity, and it will benefit from higher float voltages.

Optional setup for frequent and long outages.

In case of frequent and/or long outages It could be theoretically possible to use a dedicated battery charger to charge the battery using a bulk charing constant current phase, a topping charge, and a trickle charge phase commonly used in “intelligent” chargers, in addition to the SC-120W. This would be done by connecting the battery terminals to such a charger, and a diode between the SC-120W and the battery, to prevent the SC-120W from backfeeding the battery, that is, current flow would be restricted from going from the SC-120W to the battery, only allowed from the battery to the SC-120W. Charging would be performed only by the added dedicated charger. Such a charger should be a model that allows for continuous use.

Overall, it would serve to protect the SC-120W from having to deliver large currents to charge the battery up to its nominal capacity after a long outage where the battery is deeply discharged.

The downside of that setup would be the additional voltage drop from the diode between the SC-120W and battery while operating on battery, as it could bring the voltage into levels below 12V

Ideally, the voltage setting of the SC-120W would have to be setup so that it is more or less the same of what the charger voltage is at its max value minus the voltage drop of the diode, and that is program-phase dependent, To complicate issues more, the voltage drop of the diode also depends on current. Some chargers output high voltages for short periods of time in the equlization phase. Best is to have a good charger datasheet that explains well how the charging is done and allows for a good amount of configuration. Having voltages more or less equal could help in the regulation of the SC-120W. As the circuit of the SC-120W is not published, it is hard to say what would be the behaviour of the unit if it is setup at, let’s say 13.2V and a charger outputs 14.4 V, the SC-120W would then see 13.8V (accounting for the voltage drop of the diode added between the battery and the SC-120W), while its setpoint is at 13.2V

I encourage you to perform extensive and careful testing if you wish to go this way.

Behaviour while in UPS mode.

Voltage regulation behaviour testing should also be performed when the unit is on battery (without AC). Unfortunately, I lapsed to note data from this part in my testing protocol. If I remember well, the voltage seen by the load is then the battery voltage minus a diode drop. That is, the SC-120W does not perform dc step-up conversion to keep the voltage at the setpoint of nominal operation (on AC) Again, this should be tested, as designs may change over time. I will update this material when able to perform the test.

Behaviour on battery power loss / cold start mode

One peculiarity of the unit : If the battery connection is lost during a power outage, and is subsequently regained while the outage remains, or if the unit is connected to a battery to supply a load while it has no AC power, the load will not come online again by itself. There is a little button near the voltage potentiometer that will force supply power to the load, this function is known as a form of cold start.

It may also be possible that this function activates if the battery voltage falls under a certain level, to protect the battery. but I have not tested it.

LED indicator strip

There are three LEDS, on a strip connected to the unit through a ribbon cable. This is practical for industrial front panel installation.

  • One shows that AC is available (red LED)
  • One shows that DC power is supplied to the load (green LED)
  • One shows that current is flowing between the SC-120W and the battery. (red LED) It glows brighter under high current conditions. I suspect this is the “battery is charging” indicator. When charging current settles down, the red LED light slowly turns off. I have not tested if this LED also glows when high current is drawn from the battery.

A final warning.

The battery cable of the SC120W does not have inbuilt fuses. It is always required to implement fuses on battery links. Add a section of cable with male spade connectors (the SC120W battery terminations use female spade connectors) with a fuse in it.

Overall rating.

The unit is sturdy and the UPS function performs very well, (for over a year) and it charges/discharges the battery as advertised. There is 0 downtime during switching events. The only down sides is that is a CV control mode only (no constant current charging phase) and that the system load voltage is related to the float voltage of the battery, Given the price of the unit that’s overall worth it.

There is also a 180W unit. This could also proove a good choice to provide battery backup to larger DC loads like NAS appliances, Beware, some units require a dual +5V + 12V psu, replacing that PSU by the SC180W would require two instead of one, higher rating DC/DC step down converters.

Thermionic VST3

What is Thermionic?

Thermionic is an FX VST3 Plugin.
It is waveshaping in nature by applying a custom dynamic range compression followed by a single-ended triode amplifier.
The single-ended triode circuit uses a SPICE algorithm to solve the circuit state.
It can do both very low THD and heavy distortion/waveshaping effects.

The idea for this plugin came after designing this waveshaping compressor in LTspice. Then I thought that it would be cool to add a final triode stage.

Custom compression :
The compression applies a linear-to-log conversion with an additional coefficient (parameter k) that alters the transfer function to provide varied waveshaping effects.
Also, the knee effect can be tailored to provide distortion at the knee level. It is roughly the digital implementation of the LTspice knee-breaker compressor that I discussed in a previous post.

As for the triode implementation, it uses a classic 12AX7 single-ended configuration circuit. with AC coupling at the input and output stages for the default preset, with additional presets available.
It features a cathode bypass resistor and capacitor, grid current emulation, and Miller capacitance.

The intrinsic triode parameters are based on Norman Koren’s model and allow the emulation of other models of triodes.
Grid current is based on Rydel’s model.
Non-realistic Vact parameter is added to start grid current distortion at an arbitrary voltage input level, disregarding biasing.

Since the state-based nature of the circuit, some parameter combinations may result in divergence and are non-recoverable, but it is rare, Some parameter combinations also require more CPU time to find a solution.
So, it is advised to tread carefully when modifying parameters.

Saving a preset / restoring a preset is possible and the simulation should start all right but with a delay of a handful of seconds to obtain convergence.

abstol defines the SPICE solution tolerance and improves quality at the expense of more CPU load.

The plugin also features an experimental routing of the gain reduction to some triode parameters.

Additional features :

  • Oversampling from 0x to 3x.
  • Auto Make-up gain on/off with trimming.
  • Compressor / Triode bypass.
  • Compressor Stereo Linking On/Off.

Beta Caveats / Bugs :

  • Not all parameters are suitable for automation.
  • In some cases, when CPU load is high, switching on/off checkboxes and a StringListBox parameter, (for instance, Oversampling, GR Routing) may not be taken into account. If that happens, Please stop audio processing and change the parameter state, and then resume audio processing.

Tests :

To ensure a quality product, we will try to test the plugin in the most varied configurations, including a wide variety of OS and Hosts configurations.
Most of the tests to date have been done on Renoise 3.41

We also plan to enroll users for beta testing. Nothing is better than field experience.

Release Date :
To be announced,
Probably June to August 2023.

Minimum OS/Software requirements :

  • Windows 8 to 11.
  • x64 OS.
  • A VST3 compatible host.

What about Mac users?

We envision a plugin for Mac users, but we need more time to port some functions.

Will there be a Demo version?

There will be a demo version available on our portal soon, to start the beta field testing phase. For now, there is a youtube video with a demo of the product for waveshaping applications.

Some Screencaps :

Triode operation, Default Settings, Harmonic content.
Triode operation while drawing grid current
WaveShaping operation.

For a more dynamic presentation, check the video :

Thermionic VST3 Demo Video

The knee-breaker

Analog compressor designed from scratch

NOTE: the download link is at the end of this article.

NOTE : A VST3 plugin was made implementing the knee management part of this compressor, more info here :
https://www.skynext.tech/index.php/2023/05/31/thermionic-vst3

This post demonstrates a VCA-based compressor with a variable knee.

The methodology used to design this compressor was to draw no inspiration from existing VCA compressor design and rather start from compressor theory alone.
The fact that I had no previous exposure to VCA designs and started with no previous experience in analog compressor design is a double-edged sword.
On the one hand, it could end up with novel and unexplored means of achieving certain effects and sound coloration,
On the other hand, it could lead to questionable choices or overly complicated designs.
There is also a big risk of getting a design that works well “in silicon”, that is, running well in a SPICE simulator, but would offer subpar performance in any realistic
implementation. I am mainly thinking about SNR characteristics.

An understanding of theory is indeed required to build a compressor that achieves its intended goal, dynamic range compression.
The following paper is a good starting point :


[1] “Digital Dynamic Range Compressor Design— A Tutorial and Analysis” from Giannoulis, Massberg, and Reiss.

https://www.eecs.qmul.ac.uk/~josh/documents/2012/GiannoulisMassbergReiss-dynamicrangecompression-JAES2012.pdf

Our design however diverges quite fast from the standard gain computer architectures provided in the paper.
Figure 7 of [1] introduces the basic configurations of gain computers and sidechain detectors.

Our design is a variant of the log domain detector, (7.c), in which the level detector comes before the gain computer.
Since our level detector and A/R stages work on a log signal, and our A/R stages, being RC single pole filters exhibit exp. behavior in transient response,
which means that the overall A/R envelope with respect to the linear domain is also linear (log and exp cancel out to give a linear envelope).

This is one thing.

The next divergence from a classic compressor is the knee implementation.
Here we embarked in a really experimental direction.
Usually, the knee is an amplitude band of the signal, in which the gain computer does not apply the full ratio. This bandwidth is called the kneewidth, centered around the threshold.
At threshold – knee, the compressor applies a unity ratio, which means no compression.
at threshold + knee, the compressor applies full ratio, that is the set compression.

The goal of the knee is to provide a smooth transition between these two extremes.
When using high ratios which get close to limiting range, that is particularly useful.

Figure (4) of [1] shows the piecewise definition of the gain computer.

  • Yg is the output sample
  • Xg is the input sample
  • T is the threshold
  • W is the kneewidth

$$ 2(X_{g} -T) < \frac{W}{2} \Rightarrow X_{g} $$

$$ \left |2(X_{g} -T) \right | \leqslant W \Rightarrow X_{g} + \frac{(\frac{1}{R} – 1)(X_{g} -T +\frac{W}{2})^{2}}{2W} $$

$$ 2(X_{g} -T) > W \Rightarrow T + \frac{X_{g} -T}{R} $$

(1) (2) and (3) form the piecewise definition of gain computer function.

This function in the knee zone uses a quadratic function to make the junction smooth.
Making a smooth junction between two secant lines is a Bézier curve.

Our approach is experimental in the sense that it uses :
the sidechain signal bounded between -W/2 and W/2 as the ratio modulator, further, it applies an attack/release to that bounded signal.
To achieve proper results that A/R settings should not diverge much from the main peak detector A/R, or the time constants will lose correlation and the knee may be applied outside of the knee range or not at all. (ganged potentiometers would be required).

Furthermore, the processed knee control signal after the A/R range is passed through a tanh() function cell using a BJT differential pair.
The resulting control signal does not guarantee smooth branching between the two gain lines and may induce undershoot/overshoot.

We expect it to introduce distortion in some cases and make the knee setting a complex task, however, it is interesting from an FX compressor point of view.

Most of the circuit complexity is found in the processing of the tanh cell signal to normalize it so the compressor acts as a compressor and does not expand the signal,
and also provides no more than compression at ratio levels at the upper exit of the knee width zone.

Description of the knee control signal path.

As said before, we clip the log detector signal centered around the threshold between -w/2 and w/2.
Then we apply a similar A/R envelope. Then we use U22 opamp to control the gain of the input signal to the tanh() cell. A stronger signal will result in a tanh output being more “square”

The output of the tanh() cell is then passed to a normalizer using a VCA824 (U41) that ensures the signal “tanh_norm” is constrained : “vratio_buf” < “tanh_norm” < 0V )
This assumes that the previous tanh() cell signal is well behaved : -0.5V < “tanh_out” < 0.5V
if not, the following clamping stage will take care of unwanted excursions.

For normalization, we use a VCA824 IC to perform that function. This VCA is a linear domain VCA. It is designed for HF but should work well at audio frequencies.
However, it would incur some additional costs in the final product. There is also the issue of input/output voltage offsets that require calibration.

Then we have a clamping stage for supplementary protection :

This clamps “tanh_out_gain_inv” output positive excursions to the “- vratio” level (positive value).
positive excursions should not happen unless the tanh() cell current sink is above 247µA and the signal entering the tanh() cell makes the cell saturate.
If that were to happen, “tanh_out_gain_inv” would go above “-vratio”.

Finally, We added a bit of extra precaution so that gain voltage “tanh_out_clamp2_clip_buf” fed to the U23 VCA824 never goes above 0V.
tanh() normalization and clamping action is not perfect, as well as U41 VCA824 always has some offset despite calibration, so we clip above 0V here.

The result is our modulated ratio gain signal. We use another VCA824 (U23) to apply this ratio to gain_before_ratio_inv as a part of the gain computer architecture.
The rest of the gain computer is a standard implementation.

Before the gain control signal is sent to the THAT2180 IC, it needs to be normalized so as to apply a 20(log10(x)) transfer function.
This allows a simpler calibration when doing measurements.

A gain correction factor is done by U37, taking into account the gain factor of the THAT2180 and the transfer function of the diode linear to log signal converter that is situated at the start of the sidechain, after the full wave rectifier.

We use a high current, low output impedance op-amp for U37, as the THAT2180 requires a low impedance gain control signal.

Note that the threshold stage setting also uses a linear-to-log converter instead of a log potentiometer. The idea was to thermally couple the matched diodes D4 and D8 so that when there are temperature fluctuations, the threshold does not change much.

Use of the model :

Set the parameters.

“threshold” at 1 is -oo dB while 0 is 0dB
“kneewidth” close to 1 is 0dB while 0 is +inf dB
“ratio” is the compressor ratio.
“attack” and “release” are resistance values of the attack/release stages.
Higher attack resistance value -> faster attack (attack capacitor charges quicker)
Higher release resistance value -> slower release (release capacitor discharges slower)

Choose the input source signal

Edit the B1 voltage source and specify the voltage source to use. the “1V” voltage source uses a low frequency (20 Hz signal) that allows seeing the action of the attack and release settings. there are also pulse/square waveforms, a triangle, and one using a wave file as input, do not forget to set the input and output waveform filenames to your requirements.

Load the waveform settings.

There are three waveform settings :

  • compressor_v0.9_monitor.plt: to compare input signal with compressor output.
  • compressor_v0.9_tanh.plt: to inspect the knee computer.
  • compressor_v0.9_gain_computer.plt: to inspect the gain computer.

Caveats / possible improvements :

  • The input AC coupling to both the sidechain and THAT2180 is sketchy. it could generate phase problems.
  • There is no makeup gain in this circuit.
  • There is no balanced input to unbalanced converters, nor unbalanced to balanced output.

However, adding these would slow even more the simulation which is already really slow on a dual Xeon E5-2430v2 ( between 50 and 100µs/s) When using a wave file.
Expect 2.75 hours for 1 sec of simulation at this rate.
However, using standard test bench inputs gets a much higher simulation rate. (in the 2 to 3 ms/s range)

  • The knee control signal (between U33 and U46 stages) is a bipolar signal, which means we have to discard the diode between this stage, to allow the signal negative excursions, This stage is no more a decoupled attack/release stage but simply a first-order LP filter. The time constant however is determined by the {release} parameter. and is the same as the A/R signal stage in our model.
  • The two parallel paths with different A/R parameters (one subjected to a full-fledged A/R while the other to a simple LP filter, plus the different amplitudes due to the knee width selection give overall a different signal profile at the attack vs at the release, even when using same resistor values for attack and release resistors. this gives rise while using certain knee settings to a different effective threshold at attack vs release. The symmetrical wave shape subject to this won’t be symmetrical anymore.
  • Wide knee widths give wide knee width signal amplitudes between the two clip values (-w/2 and w/2). In the case of small widths, the resulting signal fed to the tanh_out stage may not be sufficient to drive it to saturation. A better implementation would normalize this signal based on kneewidth. For now, one has to boost U22 gain for small widths to ensure saturation. Failure to do so would make the ratio stage fail to reach its target ratio.
  • Control of tanh() saturation by means of U22/R65 has a large influence on the application speed of the final ratio, which means its effect may “take over” the attack release characteristics of the classical A/R settings.

As a conclusion, using a tanh() cell was a questionable design choice if ease of use is a prime concern, but introduces interesting waveshaping effects (if applied to an instrument, with very fast attacks and releases)

it also allows knees ‘harder’ than no “knee”. The effect can be sought as a transition from a unity ratio to a ratio more than intended before a gradual recovery to the desired ratio. The precise reason for this behavior would require an in-depth analysis of the simulation and also a practical simulation to see if that can be reproduced. It could be useful to limit fast transients since at the crossing of the threshold a super hard knee would help to bring them in check before reverting to the desired ratio

I am posting the asc without the additional required components for copyright reasons.

Simulation Data

We will now show the compressor behavior before a full calibration is done on the output gain stage op-amp final amplifier, using theoretical values.

Figure 1 : This first plot shows a step of the parameter knee width with constant tanh() gain. for smaller widths, the tanh() cell fails to saturate and the full ratio is not applied. resulting in less compression. For larger widths, compression starts earlier in our design because the main gain computer applies the A/R envelope at (threshold – kneewidth/2), to this the distortion effects induced by the tanh() cell are compounded. overall it achieves more compression at larger widths. which is quite paradoxical.

Figure 1 : stepping the knee width, R65=2k

Figure 2 : The second plot uses the same parameter stepping with a higher input gain to the tanh() cell, setting resistor R65 to 10k. The overall gain response is more or less the same for small knee widths, while larger knee widths exhibit a dramatic distortion effect.

Figure 2 : stepping the knee width, R65=10k

Figure 3, We stepped the tanh() cell input gain resistor. the knee inversion from soft to “hardcore” is clearly visible

Figure 3 : stepping tanh() cell input gain

Figure 4 : Zooming in on the previous figure to show knee inversion clearly :

Figure 4 : Effect of tanh() cell input gain. We can see knee inversions but also soft knees.

Figure 5 : We step the ratio from 1 to 10 by increments of 1 to show a more classic compressor behavior. The R65 resistor is set at 500 so the tanh() cell is not saturated. Gain reduction seems to hit a hard limit before reaching to effective ratio of 10. The compressor shows and effective ratio of 1.56 where it should be 2, 2.10 where it should be 6, and 2.23 where it should be 10. The gain computer works on a logarithmic signal, but the THAT2180 IC is an exponential control IC, that is, it expects dB input, so the result should be linear. We will have to investigate this. Part of the problem is due to the tanh() cell not being saturated but that does not explain such low effective ratios.

Figure 5 : Effect of ratio on signal, unsaturated tanh() cell

Figure 6 : Same problem. We will have to check the R26 value that sets the U37 gain used in the final gain normalization stage. Also maybe a diode voltage drop was not accounted for. in the A/R stage. The best would be to set an acceptable knee width (not too small to prevent inversions) and not too big to lower the ratio, Set a high ratio, and perform calibration by selecting a better value for R26 and then re-check compression for a ratio of 2. Fortunately, unit ratio results in a more or less identical signal which means that there is probably not any unaccounted offset in the gain/knee computers.

Figure 7: Effect of ratio on compression using R65 = 10k

Now in Figure 8, we see an adequate ratio using a knee of 0.8 but at the expense of a very hard knee and signal distortion, because of that, the signal appears to have high compression because of that offset. In reality, it is more like a higher compression obtained through lowering the threshold (which the knee setting does)

Figure 8. effect of ratio on compression. Kneewidth setting 0.8

In the end, the sensible solution was to recalibrate the final gain normalization op-amp, at the maximum rated compression level, let’s a ratio of 20, so that the (linear) ratio of the output of the non-processed signal above the threshold to the output of the processed signal above the threshold is roughly equal to 20, This is done while setting other parameters to midrange values the threshold to -6dB. This was done with the knee-width computer disconnected, and R55 connected to ground instead. In the end, we set the R26 resistor value to 35k instead of 20.8k for theory.

Next, We set the ratio back to 1 and reconnect the knee width computer since he is responsible for applying the ratio. We check that both amplitudes of the input and output signal match. The small discrepancies are probably from the VCA824’s small remaining offsets.

The process would then need to compute the ratios at step intervals and mark the potentiometer accordingly.

Figure 9: It’s much better now ! The potentiometer pot also has been replaced by an exponential pot to allow finer resolution for low profiles.

Figure 9. Effect of ratio on compression after calibration and using an exponential tap pot with linear stepping

This is the model for the exponential pot that I used :

.SUBCKT exppot 1 2 3 T=0.5
.param tap={limit(T,1m,.999)}
.param stiff={limit(stiff,1,10000)}

R0 1 3 {R*exp(-tap*ln(R*stiff))}
R1 3 2 {R*(1-exp(-tap*ln(R*stiff)))}
.ENDS

Figure 10 : For the sake of completeness, we will now step the threshold, setting the effective ratio near 4.

Figure 10 : Effect of threshold on compression after calibration and using an exponential tap pot with linear stepping of the tap

Model Download

You need these models in your library to make the simulation run properly :

  • TL072
  • LT6244 (should be included in Ltspice, check for updates if not)
  • LT1217 (should be included in Ltspice, check for updates if not)
  • THAT2180
  • VCA824

Single phase Inverter synchronization to mains using time continuous phase angle approximation with analog components

For impatient visitors, the LTspice model download is at the bottom of this post.

In our previous post we discussed the method that uses ZCD + flip-flops to extract the phase angle (angle of synchronism) using pulses whose duty cycle is proportional to the phase angle, and with a pulsing frequency of 2*f_ac, f_ac being the working frequency of the mains (grid) and inverter. Although this method is robust in the case of voltage variations, feeding pulses to our control loop required a more agressive low pass filtering strategy, and has a low gain at minimal phases angles, Overall it makes the control loop tuning harder.

So we will propose now a time continuous analog estimation of phase angle. It closely resembles to the single multiplier phase detector in the shape of the output, but does not involve a multiplier. This method is projected to be significantly more sensitive to voltage swells/sags and transients or voltage imbalances between the mains and the inverter, as it is the case for most phase detectors used in PLL. So it will involve signal normalization as well. We will try to characterize the performance of this method compared to the classic mutiplier based phase detector. Same as in the previous post, here we are discussing of synchronized inverters, not grid tied ones. As such, these inverters, perform voltage control independently of the grid conditions, that is one of the main benefits of the double conversion (online) topology, that always supplies power coming from the inverter stage at a stable regulated voltage while the grid voltage may fluctuate. On the other hand, line interactive or offline UPS perform AVR only using an autotransformer with taps to buck or boost a voltage by fixed increments. Since we have a potentially fluctuating grid voltage due to external conditions and a UPS voltage regulated at a nominal value, (not taking into account voltage fluctuations due to regulation inertia), it is important to characterize the sensitivity to voltage imbalances of the following method to assess its viability for the purpose of inverter phase synchronization.

Principle of operation

Instead of supplying the control loop a pulse whose duty cycle is proportional to the phase angle, with a postive pulse for positive phase angles and a negative pulse for negative phase angles. We supply the control loop the differential signal of V_mains(t) and V_inverter(t). That is V_mains(t) – V_inverter(t), after scaling the source signal to a level compatible with op-amps. Although it works to extract the absolute phase angle, assuming that the two voltages are of the same amplitude, preserving the lagging/leading information, that is the sign of the phase angle, requires careful processing of that signal.

Assuming a constant phase angle different than 0° and that the amplitudes of V_mains(t) and V_inverter(t) are the same,

We can see that the V_mains(t) – V_inverter(t) changes sign when V_mains(t) = V_inverter(t), although the lagging/leading status is still the same. That is why we need to switch the V_mains(t) – V_inverter(t) signal to -(V_mains(t) – V_inverter(t)) when V_mains(t) = V_inverter(t), to preserve lagging/leading information.

To encode the instant where V_mains(t) = V_inverter (t) using a basic sine to square circuit, we will feed the scaled down sum signal, (labeled ‘sum‘ in the schematic) V_mains(t) + V_inverter(t) to a comparator to get a square wave signal. The rising edge will happen at zero crossing going upwards of V_mains(t) + V_inverter(t), The falling edge at zero crossing going downwards. The points where V_mains(t) = V_inverter(t) will sit firmly at the middle of each HIGH or LOW levels time intervals. The resulting square wave signal is labeled ‘sum_sq’ in the LTspice model.

To establish a processing logic, We will also need to convert the difference signal, labeled ‘difference’ in the schematic into its corresponding square wave signal. This resulting signal is labeled ‘difference_sq‘ in the LTspice model. Note that the difference_sq signal switches polarity, that is, goes from RISE to FALL or vice versa at the points where V_mains(t) = V_inverter(t). More precisely, it is rising at V_mains(t) = V_inverter(t) when both V_mains(t) and V_inverter(t) are positive, and falling at V_mains(t) = V_inverter(t) when both V_mains(t) and V_inverter(t) are negative.

We used the LT1716 comparators for the ZCD sine to square conversion. It also conditions the square signals to 5V logic levels. It is tolerant to an input going down to -5V in relation to negative rail, here GND, while still outputing a valid 0V output in this case. This information is available in the datasheet.

Next we will establish a truth table for the above two signals.

TRUTH TABLEdifference_sq RISEdifference_sq FALL
sum_sq HIGH10
sum_sq LOW01
D flip-flop truth table

Note that we compare an edge signal to a level signal, for this edge triggered logic, a D type flip-flop comes handy. You may also ask why we need this convoluted logic, well it is necessary in order to preserve the leading/lagging information. In order to do that, we will need an additional logic stage between the above resulting signal, labeled ‘dflop‘ in the model, and the difference_sq signal. This time both signals are levels, so to establish the following truth table we will simply use a XOR gate.

TRUTH TABLEdifference_sq HIGHdifference_sq LOW
dflop HIGH10
dflop LOW01
XOR gate

The resulting signal will condition the state of the SPDT switch IC, the ADG333A IC is suitable for this application. The silicon SPDT switch will switch the output between $$ difference $$ and $$ \overline{difference} $$ input signals.

And that’s how we get an approximation of the phase angle, preserving the leading/lagging information. Note that the logic signal coming into the silicon SPDT switch not only has the result of switching polarity of the difference signal when the phase angle goes from leading to lagging and vice versa, but also performs rectification of the difference signal.

To better illustrate the action of the whole signal conditioning logic, we provide the following screen capture :

phase angle between inverter and mains oscillates between -90° and + 90° centered around 0°
Logic of the continous phase angle approximation signal conditioning block

Now that we have our proper phase angle approximation signal, it is time to feed it to the control loop.

Remember from our previous post that, assuming same frequency and voltage for both signals, and a constant phase shift or a phase variation frequency that is negligible compared to f_ac :

$$ (1)\hspace{1cm} \left | \Delta \varphi \right | = 2arcsin(peak( \frac{\left |V_{mains}(t) – V_{inverter}(t)\right |}{2V_{max}} )) $$

with peak() defined as the function that returns the peak value as a step function over the time range of interest defined below.

Note that for : $$ (2)\hspace{1cm} \left | \Delta \varphi \right | \ll \pi $$

$$ (3)\hspace{1cm} \left | \Delta \varphi \right | \approx peak( \frac{\left |V_{mains}(t) – V_{inverter}(t)\right |}{V_{max}} ) $$

Being sinusoidal in nature, it follows that for a time interval $$ (4)\hspace{1cm} \left [ t_{1} , t_{1} + \frac{1}{2f_{ac}} \right ] $$ or multiple thereof,

(3) is a linear relationship because $$ (5)\hspace{1cm} peak(k\times a(t)) = k\times peak(a(t)) $$ provided that (2) is true. Note that for the ZCD discrete phase angle method of our previous post, there is a linear relationship over the whole [ -pi , pi ] domain.

The main difference then lies into the LP stage filtering response of our control loop between a variable duty cycle bipolar square wave signal with 2*f_ac frequency and a bipolar sinusoidal signal with rectified sine harmonics at 2*f_ac frequency.

Phase angle control loop

We reused at first the phase control loop from our earlier post design :

https://www.skynext.tech/index.php/2023/01/16/single-phase-inverter-synchronization-to-mains-phase-using-the-zero-crossing-method-and-proportional-derivative-control-with-analog-components/

Since this post, it has been updated with an additional integral term to get a PID control loop.

This loop already gave relatively good results (phase angle < 0.75° for most disturbances in our simulation bench). We used it too gather data in the new phase continuous model as a reference for improvement.

Then, we optimized the loop design to get a better phase response. For this, we got rid of the butterworth filters after the integral and derivative stages, as well as a tuning of the integral cutoff frequency and the derivative peak response frequency. We will post both results here as well as the Bode plot of the new control loop.

Voltage imbalance sensitivity

Voltage amplitude imbalance between mains_scaled and inverter_scaled has effects on the diff_out signal that are mostly characterized by a reduced sensitivity to small phase angles. The signal shows a larger DC bias, which swamps the response to angle variations.

The leading/lagging transition response seems less affected, the system being able to detect the transition in small phase angle oscillations, even in the presence of a moderate voltage imbalance.

Let’s discuss the possible mitigation strategies of the voltage imbalance sensitivity.

For the purpose of phase synchronization outlined above, the inputs of the control system are :

  • Inverter voltage sensing coming from an isolation transformer on the output of the inverter.
  • Mains/grid voltage sensing coming from another isolation transformer

Both of these inputs could also be used for voltage (amplitude) sensing. Inverter voltage sensing is already used for inverter voltage (feedback) regulation. If we wish to compensate the voltage imbalance for phase synchronization, we may need to sense both.

Voltage amplitude sensing methods usually implement peak detection using smoothing capacitors and a full bridge rectifier.

Inverter voltage is dictated by the inner voltage/current control loop and possibly an outer control loop. It is subject to a certain amount of inertia. Moreover, set/regulated voltage may well be different than the nominal 240V AC.

Mains/grid voltage is dictated by the grid. We also have to take into account the serial impendance of the transmission line and that of the 10kV/240V utility transformer. These will produce a voltage drop dependent on the load, and account for a large portion of voltage variation during the day.

If, for whatever reason we wish to implement the proposed method above we would need to get rid of the voltage amplitude difference.

  • Either we establish the mains voltage as a reference, and make the inverter follow it, by controlling the amplitude of the independently generated sine wave reference of the SPWM modulator. In that case, it defeats one of the main purpose of an inverter, specially for online (double-conversion) UPS, which is voltage stability independently of the grid.
  • We could also use the mains voltage as a reference to the full extent, after scaling it down, by using the mains voltage waveform as the sine wave reference used in the SPWM generation, in that case, the inverter also follows frequency and phase of the grid as a bonus, which render the whole synchronization issue of the present article moot. The downside is that the inverter output is now subject to all disturbances of the grid, including transients, noise, etc… if adequate filtering is not provided. The inverter now works as a class-D amplifier.
  • Third option, we establish inverter voltage as a reference, and make the mains (scaled down voltage input) follow the inverter voltage in terms of amplitude. Since we have no control on the voltage from the grid, the only method that seem plausible would be to perform AGC (automatic gain control) on the sensed mains voltage to make it follow the sensed inverter voltage.

The later is not without problems though. We predict that there may be quite a high amount of crossover interaction between the phase/frequency control loop and the voltage/current control loop, making tuning of both difficult. Let’s try nevertheless.

Implementing AGC on mains voltage sensing

Since an inverter voltage control loop usually implements voltage sensing for its output using a peak detector (with attack/release control), And that doing the same for the mains voltage is also usually a requirement, for instance, to detect voltage sags/swell that go beyond the AVR capability, or simply for mains blackout detection, it seems that it would not cost much to at least try to implement an AGC for the goal of phase angle synchronization using the peak detectors outputs as differential inputs to generate a voltage control signal based on the voltage error that will be subsequently applied to a VCA. The VCA will perform AGC on the scaled mains voltage signal to keep it at the same amplitude that of the scaled inverter voltage. Then phase angle measurement can be performed without worry about the effect of amplitude imbalance.

The VCA would not need to have fancy requirements. It does not need high bandwidth, since it will work on a 50 Hz signal. It does not need high dynamic range, since it will operate on a mains voltage plus/minus 20% (worst case scenario) deviation from the nominal 240V AC. (Mains voltage is required in Europe to stay in the plus/minus 10% range from the nominal 240V AC.)

However, It would preferably use linear voltage control of the gain. That is to ease the loop design and tuning.

Voltage transient filtering (or what remains of it after the TVS upstream) could be achieved by tuning the attack potentiometer of the peak detector stage. However a compromise should be found between a good transient response and a good voltage following response so as not to introduce too much delay. This is not an easy task.

Given the requirements, a TI VCA824 IC seems a good choice. Other options although not tested would be to use an OTA like the LM13700, Finally we could also use an audio VCA like the THAT 2180x series, but it also OTA-like since it sources/sink current at the output, so either a resistor or better a current to voltage op-amp block is needed at the output. However the THAT 2180x is an exponential (dB/V) voltage controlled device, Whereas the VCA824 IC is linear (V/V). An advantage of the THAT 2180x is that it features a 0dB gain at 0V center point. It is not the case for the VCA824, Where the unity gain is closest to 0V gain control for a 2V/V max gain setup (dictated by the Rf/Rg feedback resistor setup). Even with a 2V/V gain setup the unity gain point is not exactly at 0V (at least in our setup). But this is not that much of a problem since there is a control loop for amplitude that takes care of it. Other issue encountered with the VCA824 IC is that we had to correct input and output offset voltages using voltage dividers at the signal input and output as shown in the datasheet. Using AC coupling for that purpose is a big no no since it would introduce delay. Finally, there is the cost issue. VCA824 is expensive and its features underutilized since it is tailored for HF/VHF use. But it works well for VLF like 50 Hz too. Finally, there is the issue of dynamic range. VCA824 can’t take much more than ± 1.6V at input, and goes sensibly lower than ± Vs for the output. Here Vs is ±5V (rail to rail) and this is the max for safe operation. To get some operational margin for voltage sags and swells, we setup max gain at 3V/V, and the whole setup works so as to obtain a normalized 1V amplitude mains signal, whatever the voltage sag/swell condition is. We expect the setup to be more sensitive to noise because of the reduced signal amplitude that is fed to the continuous phase angle measurement logic.

Amplitude control loop to get a normalized mains signal

Amplitude disturbance

For now, we only considered single tone FM disturbance of mains grid voltage. We still have to tackle amplitude disturbance like fast voltage transients with a clamped profile (from the TVS action), temporary overvoltages/undervoltages (from load rejection / load connection events in a generator setup), and slow voltage daily/hourly variations due to load profile change across several utility subscribers sharing a 11kV/230kV transformer.

First we will test the performance with a static voltage deviation from nominal 230V and see how the AGC performs, and how the whole loop behaves.

<to be continued>

Harmonics Disturbance

This is the hard part. It is expected that with a good voltage following characteristic, the whole loop would also somewhat track harmonics from the grid. Our goal would be to track the phase and frequency of the fundamental, not the harmonics ladden signal. We could think that filtering the mains signal would be a good idea, However we would need a really flat phase response (like those of Bessel filters), and even with that, we would need to compensate the delay with something like an all pass filter tuned to bring the 50Hz signal to a 360 phase or multiple thereof. That would introduce phase problems for frequencies other than 50Hz. Moreover, Bessel response is inadequate to filter the third harmonic sufficiently since it is so close to the fundamental. We could use a Butterworth LP filter but phase response issues would be even worse with each increasing order. We could think of a really good rejection of harmonics with a resonant filter, but that would be the absolute worse of the worse in terms of phase issues. Harmonics rejection is at the current state of analog filter technology an intractable issue in our opinion and would be better tackled in the Z-domain. Comment if you disagree.

Nevertheless, we added a harmonics disturbance setup in our model with 3rd,5th,7th,9th and 11th harmonics setup with amplitude (in % of fundamental amplitude) and phase (for each harmonic) to characterize the performance. At this point, the equation (3) is unsuitable, unless we compare the output sine reference to the fundamental of the harmonics disturbed signal.

Simulation Model

The simulation model includes the ASC Ltspice file with all packages dependencies (asy,sub,lib) in the same folder. There should be no need to tweak inside the file for absolute paths as they have been removed. No non-standard diodes, fet, bjt are used so there should be no need to add lines in the respective files (such as standard.dio or standard.bjt)

This model only models the PLL, not the full inverter. It’s goal is to generate a synchronized sine reference from mains voltage, and be tolerant to voltage sags/swells, frequency variation as well as harmonics. Harmonics should be rejected in the sine reference as much as possible.

Recently I added a block to simulate ADC operation with with sample time quantization and amplitude quantization to more accurately simulate an AD7366 ADC.

It includes a test bench block to simulate :

  • Amplitude disturbance
  • Frequency disturbance
  • Harmonics disturbance
  • Initial phase angle

It also includes the PLT files for plotting.

More information available in ____README____.txt inside the zip archive.

Have a nice day !

Single phase Inverter synchronization to mains using the zero crossing method and proportional / derivative control with analog components.

This is an introduction on inverter phase synchronization. The simulation files are included at the end of this post.

There are several digital and analog control methods to meet this goal.

In no particular order, we have DFT, KF, WLSE, ANF, KALMAN, PLL,FLL and ZCD. Most of them are documented in the digital (z-domain). A few only are easily implemented in analog.

We will discuss the easiest method, which is the zero crossing detection method, (ZCD) and assume that the inverter is not grid tied, simply synchronized.

Grid tie operation designs are diverse and fall into the grid following, grid forming, or grid supporting designs. This design is not intended for grid tie operation. These designs will be the object of another post.

The application goal, here, is mostly to have the inverter supply a voltage that is synchronized with the mains phase to enable seamless switching with an ATS that is external to the UPS, or inside the UPS unit independent of the technology used (offline,line-interactive or double conversion)

We will provide a hybrid analog/digital LTSpice model for phase synchronisation using the ZCD method.

It is “hybrid” in the sense that the inverter reference sine used in the modulation is phase modulated through a behavioural voltage source, that is more or less equal to what a DDS IC would do, but in a ideal manner since it comes without any quantization noise in LTSpice.

This model will be further updated. Note that phase synchronization using the ZCD method is performed using fully digital means in commercial ASICs. Providing a partially analog method is useful however for teaching analog control and for certain niche cases where the inverter SPWM generation and control feedback cannot be fully automated in the digital domain. (like using an Arduino instead of a fully capable DDS platform), In these cases, offloading part of the control loop to analog components is an option. Generally, fully analog control is less and less used except for simple feedback like in SMPS.

But there are still niche uses, for instance, an environment subject to ionizing radiation where hardening the ASIC is not possible, would be more robust in analog but that would require a fully analog control loop.

How to get phase difference between mains phase and inverter phase using ZCD the analog way ?

The analog ZCD method translates a sine wave (here, the output of the inverter or that of the mains power) into a square wave signal. the rising edge of the square wave signal happens at the upward zero crossing of the phase, and the falling edge at the downward crossing of the phase.

ZC sine to square wave conversion is done both for the mains and inverter phases. This is done using an op-amp comparator without feedback for each phase. The output signal is a square wave with rail to rail voltage levels.

Then, the two square signals are compared using two D type flip-flops, giving outputs pulses widths that are equal to the absolute phase difference information. (it outputs time information, not an angle value)

The method is explained in “Phase measuring circuit with leadlag indication” by Forrest P. Clay Jr. a 1992 electrical engineering paper.

https://sci-hub.ru/10.1119/1.16908

This method preserves the phase lag/lead information. One flip-flop provides HIGH output in leading conditions, While the other provides HIGH output in lagging conditions. Fundamental pulse frequency is the same as the mains and inverter frequency (assuming that both have a frequency deviation that is negligible compared to the nominal frequency) that is, 50 Hz in the model.

Phase difference detector circuit used in the simulation

Then, the output from the flip-flops is given to an op-amp substractor that generates a bipolar signal of the phase difference. Positive pulses mean leading while negative pulses mean lagging. Care must be given to the resistors tolerances (1% or better) in a substractor to minimize common mode interference, and a suitable OpAmp for this kind of use is prefered.

This signal is then low pass filtered to remove edge induced discontinuities. Note that usually the mains frequency slew rate is really slow because of the huge rotational inertia of all generators creating the mains distribution network and all regulation mechanisms in place. So it is not a problem to have a filter with a very low cutoff frequency. On the other hand, if the inverter were to synchronize to an islanded generator, that would be a whole different scenario. It is outside the scope of the current article. For these scenarios, other synchronization methods exists, and we named a few at the beginning of the article.

The filter used is an analog 3rd order butterworth LP filter, to get a sharp rolloff. The first stage has a quite high corner frequency, in order to minimize filter phase effects at low frequencies. Its goal is to minimize rising and falling edges coming from the ZCD and flip-flops well enough for the differentiator stage not to complain.

We then get a smoothed phase shift signal $$ \Delta \varphi $$ . This is fed to a differentiator op-amp setup. Its role is to generate the $$ \frac{d(\Delta \varphi )}{dt} $$ signal used further in the control loop. Note that because of processing this signal has a delay, so our notation is a little it abusive.

This signal is further processed using a second 3rd order butterworth LP filter, with a corner frequency way lower than the first butterworth filter. This gets rid of the spikes in the signal. The corner frequency is around 1Hz.

This concludes the generation of the $$ \frac{d(\Delta \varphi )}{dt} $$ processed signal that will be fed to the control loop of the inverter, for the derivative term.

In parallel, we need to get the proportional term. This will use a single butterworth 3rd order LP filter that branches just after the substractor. This will generate the proportional term also fed to the control loop of the inverter. This filter has a lower corner frequency compared to the first LP filter stage used for the differentiator branch.

Note that the final butterworth filter of the derivative signal branch has been slightly tuned out from its canonical form to get an appropriate control loop frequency/phase response.

Other than that, the remaining filters are quite the same. The differentiator has an added C2 capacitor to filter high frequency terms and provide less oscillation.

These two signals (proportional and derivative) are then factored with their respective gains (both are the same in the simulation) and fed as a sum to the behavioural voltage source of the inverter using the phase term of its function.

Note that this is a simplified model of the inverter stage. A more realistic but computationally intensive model would control the sinusoidal reference of the SPWM stage of the inverter, and inverter output would be fed to one leg of the phase difference detector. This would integrate the whole SPWM inverter model to this simulation. Note that this simulation do not include RLC loading of any of the phases. Also, this model supposes that the mains and inverter AC voltages are the same and stable at AC RMS = 230V.

One advantage is that the ZCD method is quite tolerant to voltage variations, compared to methods that are sensitive to it like PLL, So It is not critical to have it factored in this simulation.

Control loop. Not shown the final mixing stage that happens in the BV sine source of the inverter

AC Analysis of the control loop

An open loop AC analysis starting from the input to the first LP butterworth filter up to the output of the sum of the derivative and proportional terms with their respective gains has been performed.

Content of the control loop AC simulation file
Bode plot of the control loop. Plain line is Gain, Dotted line is phase

The range of frequency analysis for our first inspection is 0.001 Hz to 100 Hz

The cutoff frequency is approximately 0.66 Hz

The DC Gain is approximately 43.2 dB with a flat response.

Gain margin is -3.5 dB (at f_GM = 48.1 Hz) This could be improved for stability, knowing that this frequency is quite critical being close to the 50 Hz component in the phase difference pulse signal.

Phase Margin is 9.8° degrees (at f_PM = 38.6 Hz). Phase margin could also be boosted. Phase margin stays positive below f_pm.

There is a pole around 0.66 Hz and another close to 1 Hz, barely visible in the plot.

The control loop will be further optimized when I have time. I am no guru of control loops and filters so if you manage to get an optimized model, please chime in using the contact form…

Mains Disturbance simulation

Frequency disturbance

The first goal is to characterize how tightly the inverters locks on the mains frequency that is, $$ min(\Delta \varphi) $$ and $$ max(\Delta \varphi) $$ for a given mains frequency disturbance scenario. We also used a simple function to get an idea of the magnitude of the absolute phase difference by plotting $$ \left | V_{inverter} – V_{mains} \right | $$ and look at the local maxima. note that this plot does not suffer from the delays coming from the LP filters.

Simple FM disturbance

The disturbance scenario modeled this far is a mains frequency oscillation with a parametrized slew rate and oscillation amplitude, using a simple FM modulation scheme. The peak instantaneous frequency deviation will be restricted first to ±0.2 Hz to get in line with the ENTSOE ordinary and contigency frequency deviations, that is, an oscillation between 49.8 and 50.2 Hz

Frequency Stability Evaluation Criteria for the Synchronous Zone of Continental Europe

(section 3, Evaluation Criteria)

https://eepublicdownloads.entsoe.eu/clean-documents/SOC%20documents/RGCE_SPD_frequency_stability_criteria_v10.pdf

There is also this small study from Twente University from 2005 about stability of mains tied clocks.

Accuracy and stability of the 50 Hz mains frequency

https://wwwhome.ewi.utwente.nl/~ptdeboer/misc/mains.html

Since time keeping by these clocks rely on the number of cycles of the mains period, it makes sense to calculate the phase error. This study precisely do this, measuring phase deviations and not only frequency deviations. Phase errors in a power distribution grid come from frequency instability. To compensate for phase errors, an utility company would have to precisely manage frequency compensation at regular intervals to “get back” to the theoretical number of cycles expected. The priority is frequency, not phase, and mains tied clocks are superseded by GPS. However, this anecdotal study is however of special interest since we are dealing with both frequency and phase adjustments in grid synchronization. Note also that an abrupt phase adjustment in a rotational generator such as synchronous machines used in power plants would come from disastrous events such as pole slipping and/or sudden uncompensated load rejection. It should never happen on the scale of an utility grid.

As for the frequency, the slew rate for mains frequency is extremely low in ordinary and even contingency modes, So a rate of 1 Hz is already an extreme worst case scenario. Higher slew rates however happen with islanded generators, but this is outside the goal of this simulation. Given the response of the control loop, low slew rates should not pose a stability problem. However, this depends on the detection threshold of the flip-flop stage. A minimal instantaneous frequency deviation would not be catched until it reaches this threshold.

Note also that frequency deviations include stochastic noise but also predictable deviations according to load consumption and power generation imbalances. Periods of high demand typically introduces a negative frequency deviation until the power generated matches the load power.

As said before, the ZCD method is sensitive to harmonic disturbances typically introduced in non-inverter type islanded generators with low power handling capability relative to load. Thus, further characterizing the control loop for worst case scenarios would need to introduce this kind of disturbance, if one were to use ZCD with generators nonetheless.

Amplitude disturbance

<to_be_continued>

Phase synchronization from arbitrary initial phase difference

The other goal of the simulation is obviously to track the performance of phase locking from an initial arbitrary phase difference. The inverter has to lock its phase at 0° degrees phase difference from any starting phase difference ranging from -180° to +180° degrees. The performance of this locking process, that is how fast the phases converge to 0° and if the inverter experiences excessive harmonic disturbances during this process will have to be characterized.

Assuming both mains and inverter voltages are of the same amplitude, perfectly sinusoidal, and that the inverters track frequency change instantly or that the simulation is performed at fixed AC mains frequency, performance of phase synchronism can be measured through the following formula, giving the the absolute value of voltage phase difference.

$$ (1)\hspace{1cm} \left | \Delta \varphi \right | = 2arcsin(peak( \frac{\left |V_{mains}(t) – V_{inverter}(t)\right |}{2V_{max}} )) $$

Note that for $$ \left | \Delta \varphi \right | \ll \pi $$

$$ (1a)\hspace{1cm} \left | \Delta \varphi \right | \approx peak( \frac{\left |V_{mains}(t) – V_{inverter}(t)\right |}{V_{max}} ) $$

with peak() defined as the function that returns the peak value as a step function over the time range of interest defined below and

$$ (2)\hspace{1cm} V_{max} $$

the mains and inverter voltage amplitude.

$$ (3)\hspace{1cm} \left [ t_{1} , t_{2} \right ] $$

Since the ‘periodicity’ (the periodicity of the mains frequency induced harmonic component) of the function above is $$ \frac{1}{2f_{mains}} $$, that gives the optimal sample time to extract the maxima when sampling the above function (1)

The above function (1) can be simply plotted. If you need to extract maxima at sampled intervals use these LTspice directives and loop them with subsequent time intervals of $$ \frac{1}{2f_{mains}} $$ and put them into a .MEAS file, although it would need a long simulation time to make sense. For complex data analysis it is better to make a LTspice export of the data and process it with Python for instance.

.meas TRAN Vdiff_abs_norm_max MAX (abs((V(vl) -V(vn)) - V(mains))/(2*1.414*{V_ac}) ) FROM 0ms TO 10ms
.meas TRAN delta_phi_abs_max PARAM 2*asin(Vdiff_abs_norm_max)

measuring the minimum phase difference (best performance at steady state) is less trivial because of zeros in the function when the sine waveforms cross each other, therefore it would require sampling the phase difference with the above method and then analyze the resulting data for local minimums. Overall, another useful metric is simply done by averaging the sampled maximum phase difference (CTRL+click) for function (1) over a long interval, preferably equal to a full oscillatory cycle that arises from the control loop, if one is found.

Finally, performance in phase locking has to be demonstrated in conjunction with a FM disturbed mains frequency.

Note that phase locking is preferably done while keeping the waveform sinusoidal in nature during the process. Phase locking in figure 2 happens too quickly, and has the effect of producing severe distortion. The inverter should have adequate protection to not supply power during this event, only after proper phase locking is done. In a mains synchronized double conversion UPS, this could happen if the input is switched between two phases (120°shift) or after the powering on and transfer to a generator. Since a double conversion UPS always provides power through the Inverter stage with no possible downtime except a minimal one for switching to and from bypass mode, the control loop has to be tuned to take that into account. A modification of the phase control term in the sine wave DDS generator could be done and would take effect only during these initialization/switching events, for instance, using

$$ 1-\exp\left(-a\cdot t\right) $$

as a factor of the control term, ‘a’ controlling how fast the control loop locks into the mains phase.

Simulation results

Fig. 1 initial phase difference +90° , frequency disturbance : modulator freq. fm = 1Hz, modulator Amplitude Am = 0.3V
Fig. 2 Initial phase difference +180°, fm disturbance unchanged. Note the large disturbance of inverter phase during the lock process. Locking happens in less than a period
Fig. 3 frequency disturbance : modulator freq. fm = 1Hz, modulator Amplitude Am = 1V
Fig. 4 frequency disturbance : modulator freq. fm = 1Hz, modulator Amplitude Am = 1V At this slew rate of mains frequency and high frequency deviation, performance is degraded.
Fig. 5 frequency disturbance : modulator freq. fm = 1Hz, modulator Amplitude Am = 0.3

Using the ZCD method, sampling time is limited at two times the mains AC frequency. That limits accuracy of the algorithm for fast and ample disturbances. But a heavily distorted power source would not lead to any application requiring syncing into it, rendering that issue moot.

On the other hand, ZCD is quite tolerant to voltage fluctuations.

Conclusion

Although this controller is simple to implement, it suffers from steady state error due to the limited gain at DC. One option to mitigate this is to add an integral component. However, it would still suffer from delayed response to oscillations due to the butterworth filters, and cannot track fast oscillations. The ZCD+Flip Flop stage also samples phase at 1/2*f_ac, which is a limiting factor. The non linear behaviour introduced by the discrete function of the flip flops, who encode phase difference as a pulse further make the tuning of the control loop harder, with the need to analyze the impulse response of the system. However the discrete ZCD phase difference method is more robust when it comes to voltage imbalance between the two measured phases.

In a next post, we will propose a time continuous control of phase difference without flip-flops in the control loop signal path (although one is still needed in the circuit).

To get back to the conclusion on this model : Its performance level is unacceptable for grid tie operations, nor it provides the required functions and behaviour of GFL,GFM,GS grid tie topologies. That is why we limit it to synchronization for inverter standby/autonomous operation to alleviate source switching transients. But it is a good introduction on the subject. For an idea, it is closer to the state of the art for the start of the 90s or so, when digital control was not yet so widespread.

We will discuss grid tie inverters in a later post and slowly but surely move into the more state of the art technologies. It will also serve as an introduction into fully closed loop control systems, as with grid tie inverters, voltage,current quantities are intimately tied, and reactive power effects have to be taken into account.

Beware, the learning curve will be steep.

Simulation files

EGMicro EGS005 board review

The goal of this post is to analyze in detail the advertised features of the EGS005 board, and show possible modding hacks.

The EGS005 board is the newest board provided by EGMicro for single phase and multiphase inverter designs.

It is based on the EG8025 ASIC that features integrated MOSFET drivers for a full bridge configuration.

Most if not all all of the EGS005 information is also provided in the EG8025 datasheet, plus many more details! We will use the EG8025 datasheet as the reference material, but also compare them to the EGS005 board features, to see what features are restricted by the board.

The EG8025 datasheet is available on the EGMicro website, chip center section.

Probable IC orientation

Pinout analysis and IC orientation.

Based on the application schematic and components names and indexes placement, and after boosting trace contrast, it seems pretty evident that this is the ASIC orientation on the EGS005. The D4, D5 diodes and the C17 capacitor with its traces clearly shown going up to the pin, plus the 3 NC pins at the bottom and at the right side make it the only possible configuration. This setup would make the pin1 orientation dot in the product image misleading.

Differences between EGS002 boards and EGS005 boards

We will focus our attention first to the differences in features between these two boards. This takes into account only the features exposed through these two boards, not the overall feature differenciation between EG8010 and EG8025.

EGS005 has these additional features compared to EGS002 :

  • integrated MOSFET drivers
  • Test mode for SPWM output bench testing without any control loop feedback
  • overload protection (not only overcurrent hard limit)
  • Two over temperature control zones (IGBT/MOSFET and PCB)
  • SPWM signals routing swap on/off between left bridge and right bridge MOSFETs
  • AC output enable/disable through pin (basically a soft shutdown)
  • Exposed Serial interface (RX and TX), but configuration settings registers besides switching in and out of “Test mode” are either unavailable or undocumented.
  • Exposed pins for firmware update

EGS005 features that are discontinued compared to EGS002 :

  • Variable frequency output mode up to 100Hz or up to 400Hz. This includes variable frequency mode and fixed ratio V/F mode.
  • Choosing between unipolar and bipolar SPWM. Note that EG8025 supports phase synchronisation/phase shift for 3 phase mode, so the modulation scheme had to be made unique for interoperability.

EG8025 features not exposed in EGS005 :

  • Phase shift mode for AC sensing from another unit – Phase_SEL pin 12
  • AC input for phase synchronization/shift from another unit – VZC_IN pin 17
  • AC output for phase synchronization to another unit – AC_Fout pin 13
  • Multi inverter pin for parallel operation or master/slave select for three phase operation – Multi_INV pin 15

Inverter phase synchronization and phase shifting

Inverter phase synchronization is required for the following operation modes.

  • Parallel mode of operation of two or more inverters sharing a single phase for load sharing / redundancy.
  • Parallel mode of operation between one or more inverters and the AC grid, These inverters are known as Grid tie inverters. They are ubiquitous in renewable energy systems for residential or industrial use.
  • Parallel mode of operation between inverters or between inverters and AC grid, who do not share the load for redundancy (active/standby system). The phase is kept synchronized between the sources for seamless operation of an ATS (Automatic Transfer Switch). This is to limit potentially high dV/dt (and/or high dI/dt) that happen during switching when the phases are not synchronized.
  • Multiphase mode and inverter daisy chaining (cascade) of phase synchronization across usually three units, for three phase power applications, In a (master)/(slave/master)/(slave) configuration. Parentheses correspond to the three inverters.

There are several digital algorithms and analog tehcniques to implement phase synchronization.

One well known and ubiquitous method used in various electronics designs not limited to inverters is PLL (Phase locked loop). There are however other methods. This paper discusses them in detail :

Recent advances in synchronization techniques for grid-tied PV system: A review

https://www.sciencedirect.com/science/article/pii/S2352484721008118

EG8025 Phase synchronization

The EG8025 ASIC uses the Zero Crossing method. It is simple and straightforward.

It uses 2 pins for configuration. Multi_INV pin 15 and Phase_SEL pin 12 and 2 pins for synchronization data. One is an output pin, AC_Fout pin 13 the other is an input pin VZC_In pin 17.

Parallel operation mode

We’ll assume that we use two inverters.

In this mode both inverters share the load on the same phase. One unit is designed as master and has Multi_INV pin 15 pulled log to GND, The other is designed as slave and has Multi_INV pin 15 pulled high to 5V.

The master unit also has VZC_In pin 17 and Phase_Sel pin 12 pulled to GND. Since the master is the start of the synchronization chain, it won’t use an input ZC signal, nor it should shift the phase 120° for parallel operation. Applying phase shift in this mode of operation could destroy both inverters output stages !

The master outputs its phase information through the AC_Fout pin 13. This is a zero-crossing signal. It probably converts upward going zero-crossings of the AC phase to logical HIGH, and downward going zero-crossings of the AC phase to logical LOW. Rising/Falling edges should happen at the time of the zero crossings. Checking precisely the logical levels correspondence is required if this board is to work with another unit of another manufacturer supporting ZC synchronization, to prevent a 180° out of phase condition. Level shifting may be required to accomodate the slave unit.

The slave in turns gets its ZC information on the VZC_In pin 17. The path between AC_Fout pin 13 of the master and VZC_In pin 17 is isolated with the use of an optocoupler. Check figure 10.a of the 8025 ASIC Datasheet. On the slave unit Phase_Sel pin 12 is also pulled to ground while AC_Fout pin 13 is floating.

Since AC_Fout is a low impedance pin current source, it should never get pulled to GND.

Modding for parallel operation.

We should investigate the board and the EGS005 application schematic to look at the trace routing of VZC_In, AC_Fout, and Multi_INV.

Modding fo the slave unit :

  • VZC_In is pulled to GND through the R45 1k resistor. Making the VZC_In as an input as shown in figure 10.a would require soldering out the R45 resistor and supplying the signal to the exposed pad of R45 connected to the pin. This signal comes from the optocoupler voltage follower.
  • AC_Fout and Phase_Sel do not need any mod on the slave unit.
  • Multi_INV trace to GND should be cut and a patch wire soldered and connected to 5V HIGH level

Modding fo the master unit :

  • AC_Fout is floating on the EGS005 so a simple wire patch to the pin would do the trick. This wire would be routed to the optocoupler diode anode.
  • Multi_INV and Phase_Sel do not need any mod on the master unit.

For a tutorial on how to perform SMD pcb wire hooking look at :

EGMicro EGS002 board review

The goal of this post is to analyze in detail the advertised features of the EGS002 board, and show possible modding hacks.

The EGS002 board is the oldest provided by the EGMicro supplier still distributed on common Chinese reseller platforms. It superseded the even older EGS001.

It is based on the EG8010 ASIC and also features either two IR2110S half bridge drivers, or two EG2113, an EGMicro driver. Whether you get one or the other depends on the reseller. Check for comments and reviews on marketplace product page to see who’s getting what.

Most if not all all of the EGS002 information is also provided in the EG8010 datasheet, plus many more details! We will use the EG8010 datasheet as the reference material, but also compare them to the EGS002 board features, to see what features are restricted by the board.

EG8010 ASIC Features

Input DC Voltage. The EGS002 can drive high voltage MOSFETs easily. no restrictive voltage limitations on the high side MOSFETs, and it is at least ok for 400V DC input to the MOSFET bridge. Supplied design schematics show power coming through a 400V DC link PFC output.

Inverter output frequency. EG8010 can be used for fixed 50Hz,60Hz or frequency adjustable 0~100Hz or 0~400Hz output.

The EGS002 on the other hand restricts this feature to fixed frequency operation : either 50Hz or 60Hz, through jumpers.

These jumpers are exposed on the bottom plane of the board and are set with solder bridges over two pads. They do not appear to be through hole, so it would be difficult to insert header pins and use a real jumper there.

the backside exposes the configuration jumpers. top left of image

JP1 and JP5 on the board control FREQSEL0 pin 18 level (either HIGH=JP1 short for 60Hz or LOW=JP5 short for 50Hz). They can’t be short or open at the same time !!

Is the board moddable for further frequency control ? let’s see.

There seems to be two methods to apply mods. Either through hardware or through software (by serial commands). Let’s explore the hardware method first.

Variable Frequency mode modification

Up to 100Hz or up to 400Hz variable frequency operation mode selection is controlled by FREQSEL1. It seems however that the EGS002 has the FREQSEL1 pin 19 grounded in the EGS002 schematic. So it depends on the suppliers of EGS002 to create derivative boards that expose FREQSEL1.

As far as I searched on Chinese marketplaces that doesn’t seem to be the case.

PIN 19 and PIN 20 Traces Merge. right around the center of R33. Processed image to better expose the traces

FREQSEL1 pin 19 and MODSEL pin 20. seem both connected to ground in most boards available on the market through merging traces. This is in conformity with the EGS002 datasheet.

That restricts the unmodded board to 50/60 Hz Operation and Unipolar switching

Modding for tests to enable VVVF to 100 Hz or to 400 Hz would require cutting the FREQSEL1 pin trace and patching the pin with maybe AWG30 hookup wires and connect it to HIGH level. MODSEL would be still kept to GND.

JP1 and JP5 would allow to control max frequency to 100Hz or 400Hz.

Note that variable frequency with constant voltage mode and variable frequency with variable voltage both require Unipolar switching. That is why you don’t have to bother with MODSEL in this mod

Once FREQSEL1 is set to HIGH, Variable frequency mode type is enabled through VVVF pin 32. In EGS002 again, it is connected to ground. This mode would give EGS002 the variable frequency constant voltage mode by default. (without further mods)

To enable VVVF variable freq/variable voltage (albeit with constant V/F ratio) for single phase VFD applications, bring VVVF to HIGH

Pin 32 VVVF. For variable frequency mode in unipolar switching

Again, the trace after the pin may be cut if other pins do not depend on the cut trace, which may be difficult to check since some trace may hide under the IC.

Note that there are several test points / open vias on the board that can be used to patch the board with additional connections.

Then you have to expose FRQADJ/VFB2 pin to set the desired frequency in variable frequency mode through an external potentiometer in figure 8.6a of the datasheet. Voltage regulation is still performed through R23 and VFB1

In constant voltage/frequency ratio mode, you use R23 to set the nominal voltage at 50Hz through VFB1 Frequency ratio control goes through FRQADJ/VFB2 in this mode. It is a bit unclear in the datasheet.

Bipolar SPWM enable modification

Remember that you cannot use VVVF features in this mode.

The mod would require :

  • cutting MODSEL pin 20 trace to disconnect from ground and patch it logic HIGH.
  • cutting FRQADJ/VFB2 pin 16 trace/pad to disconnect it from ground and use it to supply voltage feedback as shown in the EG8010 bipolar switching application schematic. In this mode VFB pin 13 and FRQADJ/VFB2 pin 16 are supplied a differential voltage feedback. It is required in bipolar switching. .

However, all boards found on the market seem to implement the application schematic “Figure 6‐2. EG8010+IR2110S+cross‐conduction prevention logic sinusoid inverter(unipolar modulation)”

The cross conduction prevention logic is inserted between the ASIC logic outputs and the Gate drivers logic inputs. It consists of resistors and BJTs

I think that would prevent the bipolar SPWM mode from working. So you need further modding.

  • You need to unsolder Q2 Q3 Q4 Q5 and solder bridge collector and emitter pads.
  • Remove resistors R30 to R37

Using the UART to set the operation modes

The ASIC also exposes an USART interface (RX pin 4 and TX pin 5)

The data inteface looks powerful, exposing all configuration options usually set by the jumpers. Whether the override of the jumpers is properly done by the USART remains to be seen.

Note than enabling bipolar SPWM would still require the removal of the cross conduction prevention components.

It also allows for monitoring the same parameters than in the LCD Output, that is frequency, current, temperature, and voltage.

The EGS002 has the RX pin to GND and the TX pin floating, Again cutting the trace to GND could allow to access to the USART. The data interface is fortunately documented in the datasheet. it uses 2400,8,N,1 serial settings.

Dead Time Control

EGS002 implement dead time control through solder bridges at the bottom layer of the board. These are JP3, JP4, JP7 and JP8.

The EG8010 has a fixed switching frequency of 23400 Hz that makes a pulse at 50% duty cycle time duration of roughly 21µs.

That makes the ratio of dead time to pulse length quite important at 1.0 and 1.5 µs, and may impact scaling up for higher output power. The default is 300µs. That is still quite conservative.

Check your MOSFET specifications for minimum dead time requirements.

Soft start

EGS002 has a soft start 3 second feature enabled by default through JP2. I do not recommend disabling this feature.

Voltage feedback & regulation.

It doesn’t look like there are any rectification on the voltage feedback network, nor on the EGS board or outside. and the RC filter made with R4 and C4 has a small time constant of 1ms. The voltage regulation uses a peak detector and measures error in relation to a 3V reference as per datasheet.

In any case the voltage nominal setpoint is performed through the lower leg of the voltage divider using R23 (a 10K potentiometer). A slow acting external voltage control could be done by replacing R23 with a current DAC.

Bipolar switching voltage regulation

You will see in the EG8010 datasheet that the voltage divider network is more complex (it is outside the ASIC board) than for unipolar, with a ganged potentiometer R23. I also suspect missing connection dots in the schematic between R19 and R21 and also R26 and R27 and also at R27 low leg and GND.I will test it on my LTSpice model, so take that info with a grain of salt. Here is the relevant section of the schematic with the proper corrections

bipolar switching voltage divider

Overcurrent protection.

The method employed is a hard current limiter. It cuts the SPWM input to all MOSFET drivers if the output current goes above the setpoint for more than 600ms. The unit is shutdown for 16s, then it will turn on for 100 ms, check for current status and repeat that 100ms on time every 16s if the issue persists for a maximum number of 5 cycles. if the issues persists it powers off and a hard reset is required. If the issue is cleared for more than 1 minute, the state machine resets overcurrent status to nominal.

The board also uses a LM393 OpAmp to process the IFB feedback to shutdown the Gate drivers directly through their SD pins, it is much faster and failsafe than doing this only through the ASIC.

There does not seem to be any soft limiter for light or moderate overcurrent protection (that would lower load voltage for resistive and inductive loads)

A soft limiter would not help for Active loads such as AC/DC PSUs because they change their apparent impedance to compensate for voltage loss.

Monitoring and UI

The monitoring is done through a LCD interface using I2C LCD compatible modules specified in the datasheet to display basic information. Or it can be done through the serial link if you expose the pins properly.

400V DC to 230V AC pure sine wave inverter model using LTSpice.

Ltspice inverter model

Note : the above model has been updated for frequency/phase synchronization. please check :

output voltage and current waveform, 80 ohms resistive load
Disclaimer : This design uses dangerous AC and DC voltages. If you get out of the simulation domain and start prototyping be sure to use all safety precautions required when working with high voltages. You have to know what you are doing.

Besides the simulation this post is an introduction on pure sine inverter technology targeted at electronics engineers that have little or moderate experience in power electronics and inverter design.

The goal is to design, implement and prototype your own pure sine wave inverter from scratch as an educational project to get into inverter technology, this will be the object of a series of posts in the future.

For a faster design approach see the bottom of this post on how to use off the shelf inverter modules such as EGS002 or EGS005 available on BangGood and AliExpress.

To get straight into the model simulation go to the running the simulation section.

Introduction

Inverters use MOSFETS to switch a DC source with a variable duty cycle PWM signal.

The duty cycle variation in the time domain is performed at the frequency of the required output fundamental frequency of the inverter.
Usually mains frequency, that is 50hz or 60hz.

The frequency of switching, that is the frequency of the PWM signal is called the switching frequency. it is usually in the 2.5khz to 100khz range.

So, the goal is to have a PWM signal at high frequency (2.5 khz to 100khz) with a variable duty cycle whose frequency is at mains frequency.

However, The variable duty cycle frequency may be lower or higher, or can be adjusted in real time.

Applications that require this duty cycle modulation at fixed but non standard 50Hz or 60hz are mainly for the aerospatial industry.

Airplanes use 400hz. The advantage of 400hz is that power transformers are less bulky than in 50Hz.

There is also an industry need to adjust the inverter output frequency in real time. This is the market segment of VFDs (Variable Frequency Drive) inverters.

This allows to set the rotation speed of induction motors, and allow for a soft-start that does not damage the motor.

In VFD applications, not only the the frequency output of the inverter is managed, but also the output voltage, and sometimes they implement a fixed voltage to frequency ratio mode so the motor stays happy.

So, now comes the question, How to modulate the duty cycle of a PWM signal ?

This is usually done by comparing a triangle signal at the switching frequency with a reference sinusoidal signal at the desired mains frequency.

This can be done in two ways :

1) Using analog components : a sine generator IC (like the XR-2206 or MAX038) that outputs a triangle wave and another one (also XR-2206 or MAX038) that outputs the sine wave. Then, a schmitt trigger is used to compare these two signals to output a PWM signal.

2) Using digital components : the sine modulated variable duty cycle PWM output is generated by code running on MCUs, PIC, DDS IC. Arduino can do this, however Arduino has a limitation that hinders its use for this purpose, and that is dead time control. More on this later.

If you nonetheless want to experiment with SPWM generation with an Arduino, check this code to get an idea of how it works.

I recommend you read resources on Fast PWM for Arduino. It is not straightforward if you have no experience with hardware counters/timers. Check https://www.arxterra.com/tutorial-on-fast-pulse-width-modulation/ and https://docs.arduino.cc/tutorials/generic/secrets-of-arduino-pwm
for starters. You may have to browser for other resources because I could not find one comprehensive documentation for ALL modes, except maybe in AtMel datasheets, but these are very terse and quite hard to understand.

Here is the first code sample :
#include <Arduino.h>

uint16_t freq = 50; // inverter output frequency
uint32_t counter = 0;
uint16_t mod_index = 0.9; // modulation index. you will have to update this in real time for precise voltage control.
float sin_val;
const uint16_t samples_per_period = 100; 
// higher samples per period give a better looking output sine wave, less harmonics from digital aliasing
uint16_t micros_interval;

uint16_t sin_table[samples_per_period];

void populate_sin_table() 
{

  uint16_t i;
  for(i=0;i<samples_per_period;i++)
  {
      sin_val = 512*(1 + mod_index*sin(2*PI*float(i)/samples_per_period));
      sin_table[i] = round(sin_val);
      //Serial.println(int(sin_val));
  }

}

// FAST PWM PHASE-CORRECT MODE 3

void setup(){ 
   
  // Wave Form Generator: phase correct PWM mode 3, Top = OCR1A and OCR1B
  // We will output two signals, complementary, using two pins ~9 and ~10 so we need to specify
  // (0<<COM1A0) + (1<<COM1A1) + (1<<COM1B0) + (1<<COM1B1)
  // (0<<CS10) + (1<<CS11) + (0<<CS12) this is the prescaler and will dictate the switching frequency.
  // (1<<WGM11) + (1<<WGM10); and (0<<WGM13) + (0<<WGM12) are used to set the Fast PWM mode, here we use mode 3.
  // it allows a 10 bit amplitude resolution for the sine wave signal
  // check this link for a table of available modes.



  TCCR1A = (0<<COM1A0) + (1<<COM1A1) + (1<<COM1B0) + (1<<COM1B1) + (1<<WGM11) + (1<<WGM10);
  TCCR1B = (0<<WGM13) + (0<<WGM12) + (0<<CS10) + (1<<CS11) + (0<<CS12); 
  
  OCR1A = 0x3FF; // top compare value initialization. it will be varied using the sine table in the loop.
  OCR1B = 0x3FF; // same for the second PWM signal
  
  //DDRB |= (1<<PB1);

  //Serial.begin(9600);
  populate_sin_table(); // create a sine table. better use PGM write and store it in flash for a more robust approach
  micros_interval = int(float(1E6)/(float(freq)*float(samples_per_period))); // the loop wait delay between two bit bangings
  // of OCR1A and OCR1B.

  //Serial.println(micros_interval);
  
} 



void loop() { 
 delayMicroseconds(micros_interval);
 OCR1A =  sin_table[counter%samples_per_period]; // iterate on the sine table, use modulo to loop the table
 OCR1B =  sin_table[counter%samples_per_period]; // same for the second SPWM
 
 //Serial.println(sin_table[counter%samples_per_period]);
 //Serial.flush();
 counter++; // overflow not managed !!!!!
} 

There is also this wonderful code https://forum.arduino.cc/t/dead-time-between-pwm-signals/937405 for a three phase system. I did not test it but it looks serious and legit. It has the advantage of using ISR and not a loop to update the TOP value with the sine wave, generates 6 signals (so it is for bipolar spwm) but it suffers from the same dead time insertion problem however, and that is the object of the guy’s post. But it is possible with analog post processing, check the DTI section.

That is why STM32 based boards are better for an all digital SPWM generation purpose, but they are more expensive and you'll need to watch quite a bunch of tutorials to master Nucleo (the STMicroelectronics MCU IDE). It will be easier if you already master Arduino FastPWM generation, but it will require time nevertheless.

Now is a good time to learn about the specifics on how that sine modulated variable duty cycle PWM signal allows the inverter to generate a 50hz, 60hz or higher frequency mains power phase voltage.

The "sine modulated variable duty cycle PWM signal" will be now referred by its usual name in the power electronics technology as SPWM (sine PWM)

The power core : the MOSFET H-bridge

The power core of an inverter uses an H-bridge configuration because the setup of its components ressembles to the letter H.
It is one of the most common designs in the industry.

MOSFETs switch repeateadly a DC source with low source impedance (the power source) according to the input gate signal that comes from the SPWM.

That is not all. high VDS high current MOSFETS usually need gate voltages that are higher than what an MCU or an analog oscillator can generate.
The high side MOSFET gates  of the H-bridge also need voltages with reference to DC ground that are way above the levels of the logic/analog controllers.

For that reason, there is a specific family of ICs that exists and they are called "MOSFET Gate Drivers". Their goal is to bridge the gap between the logic SPWM signals and the required voltage levels (and current requirements) of the MOSFET gates.

Moreover, an H-bridge inverter has at least 4 mosfets. These MOSFETS need to be activated by gate signals at a precise fashion, like a fine tuned choregraphy.

The activation pattern is usually diagonal in the schematics. This has the effect of reversing the polarity with reference to DC GND seen by the load with each pwm pulse. Since the two SPWM signals are complementary in this design, when one diagonal set of MOSFETs has a high duty cycle it will be on and conduct a longer time than the reciprocal diagonal set of MOSFET, when this happens the output sine wave of the inverter is at a +Vo_peak. Then there is a time when the duty cycle 0.5 for both diagonal pairs, at this point the sinusoidal output of the inverter crosses 0V. then the cycle goes in the reverse direction and outout reaches -Vo_peak.


PWM Modulation schemes

I just described one SPWM modulation scheme. There are two schemes that are most commonly used.

- Unipolar SPWM
- Bipolar SPWM

To understand the difference between the two, please read now :

https://www.sciencedirect.com/topics/engineering/sinusoidal-pulse-width-modulation
https://www.tntech.edu/engineering/pdf/cesr/ojo/asuri/Chapter2.pdf

My guide is based on Bipolar SPWM.

As you may already guessed, for that SPWM scheme you need two complementary SPWM signals.
For analog SPWM generation, the original SPWM signal generated from the triangle to sine comparator is fed to a NOT gate to create a complementary SPWM signal.

The original signal will drive the Top Left MOSFET and Bottom Right MOSFET
While the complementary signal will drive the Top Right MSOFET and Bottom Left MOSFET.

Note that in an unipolar SPWM scheme, The complementary SPWM signal stays low the whole time the other one operates, and then starts doing SPWM modulation while the other one stays quiet. This switching happens at 2*f_mains,
I find it harder to generate these two kinds of SPWM signals using digital means, so I think that the bipolar scheme is better to start grasping the technology.

Note also that IC Gate Drivers usually manage two MOSFET for a half bridge configuration. So we need Two gate driver ICs
In the LTSpice model IR2110 is used, as it is quite common in the industry.

The routing between the SPWM signals and gate drivers is as follows in my LTSPICE schematic and simulation :

SPWM signal is provided to HIN input of Gate Driver 1 and LIN of gate Driver 2
Whereas the complementary SPWM signal is provided to LIN input of gate driver 1 and HIN input of gate driver 2

HO output of gate driver 1 drives the top left M1 Mosfet
LO output of gate driver 1 drives the bottom left M2 Mosfet
HO output of gate driver 2 driver the top right M4 Mosfet
LO output of gate driver 2 drives the bottom right M3 Mosfet.

If you connect the dots, you'll see it fits the requirements of bipolar SPWM modulation scheme.

The big issue : dead time control.

There is a factor that needs extra precautions because it can fry the MOSFETs and brick the inverter. In no case, the M1 M2 MOSFETs should conduct at the same time.
This also true for M4 M3. If that would happen, The low impedance DC source will short through these MOSFETs (from Vdc to GND). When driven to high gate voltages, these Power MOSFETs  have a ridiculously low RdsOn, This will generate currents way above the absolute ratings of the MOSFET, frying them if a DC breaker or another protection from the upstream DC link did not catch it in time.

For this reason, a security margin between the two SPWM signals (the original and complementary) has to be put in place, This is a delay between pulses where both SPWM signals remain low. This is called dead time.

a good article to read on that subject :

https://hackaday.io/project/3176-gator-quad/log/11741-pwm-control-and-dead-time-insertion

For digital SPWM generation, forget about Arduino to get SPWM plus dead time control using Phase correct PWM out of the box.
Better PICs or MCUs are required such as STM32.

It is however possible to apply analog treatment to a single SPWM arduino generated signal, to create a complementary signal with a dead time delay inserted before the rising edge and after the falling edge.

In the supplied schematic, it is done through a RC network whose time constant is matched to introduce the required dead time, plus a bunch of gates and schmitt triggers. This setup may be simplified, if you manage to use less components or update the model to use real world components instead of ideal ones, let me know through the contact form.

Output Voltage control and regulation.

Output voltage control of the power stage is performed basically by varying the triangle peak voltage, while the sine voltage remains the same.

The peak output voltage is roughly given by 
Vo_peak = V(dc)*V(sine)/V(tri)
Vo_RMS is then V(dc)*(V(sine)/V(tri))/sqrt(2)
V(sine)/V(tri) is known as the modulation index (mi) 
it is usually < 1

If V(sine) gets larger than V(tri), the inverter operates in the overmodulating region, and RMS output voltage is no longer linearly dependent on V(dc)
It appears in the simulation as a voltage saturation of the output.

overmodulation is outside the scope of this guide but it is explained in the above mentioned resources.

An open loop control is not realistic because of deviations from theory that assumes ideal components, and dependance on the load RLC parameters and the output filter parameters, as well of the transformer if an isolated output is required. 

So there is a need for a voltage feedback loop to the SPWM generating component to adjust the modulation index.

Passing the voltage information to the source controller has to be done properly.

Because of voltage shifts of the (V(l) - V(n)) voltage relative to DC GND even after passing a voltage divider in this unisolated design, galvanic isolation is required, and this is a safety requirement.
If the output is isolated using a mains frequency transformer, galvanic isolation in the feedback loop is inevitable.

There are three methods to create an insulated voltage feedback loop.  

- Using an auxilliary winding in the secondary of a mains transformer.
With a turns ratio tuned for low output voltages, usually such as 3.3V/2 (rms) or 5V/2 (rms) is obtained when the inverter output voltage is at nominal conditions. That gives ample room for the signal to go higher if the output stage encounters an overvoltage situation. 
One leg would be connected to DC GND, the other fed to a full bridge rectifier, smoothed, and then that voltage would be fed to an error amplifier op-amp (a difference amplifier) that compares it to a precise voltage reference. This difference signal is a negative feedback that is then used to adjust the modulation index if using full analog generation. That would happen by controlling the AM pin of the XR-2206 Triangle generator, for instance.

If the SPWM is generated with digital means, the error amplifier could not even be required.
A DAC may be enough to acquire the voltage output of the optocoupler and the comparison of the DAC output would be done against a digital reference. Then, a digital control method should be implemented to control the feedback loop (such as PID) and update the modulation index.

I digress. Back to the feedback isolation.

- The other way of isolation is to use a separate low power transformer, with it's primary legs in parallel with the load. That is useful if your design does not use a transformer at all for the power stage, so you need only a transformer for feedback. Also, transformers with auxiliary windings are specific and you may have one on hand without that feature.
Of course it should be designed for 50Hz, or 60Hz, or more depending on your output frequency, with a ratio adjusted the same way as for an auxiliary winding.


- The third method involves a rectifier, a smoothing capacitor/filter stage, a voltage divider, which output drives a voltage controlled current source.
In this design I am trying a modified Howland current pump.
This current source is required because it will drive an optocoupler to transmit the analog information across the insulation barrier.
Optocouplers are current driven.

Care has to be taken to operate the optocoupler in its region where the CTR (current transfer ratio) is linear, by setting properly the current source gain.
Then one has to chose the proper collector resistor on the output side of the optocoupler to obtain proper transfer characteristics.

It is advised to test the transfer characteristics of the Current source + Optocoupler in a separate circuit to tune it up before adding it to the design, it will make things faster to simulate and way easier.

The rest of signal processing is the same as with the transformer approach.
Either digital control if SPWM is generated through a MCU or analog if using an oscillator. Analog will need a compensation network (usually a R+C // C) network for the feedback control loop to be stable.

Note : I have yet to model this part in the supplied model. It will be the object of this post revision in the future.

However, the fact that we are dealing with an inverter and that we have to perform rectification and smoothing to extract the peak voltage information, introduces feedback charateristics that are a bit different than DC/DC converter feedback loops.

Since the feedback loop would be designed with non-monolithic components (By that I mean the that the feedback loop does not use complex IC whose internals are a black box if their full transfer characteristics are not provided in the datasheet, and that perform mysteriously during AC analysis), it is possible to perform an AC open loop frequency response analysis of the rectifier + smoothing capacitor + current source + optocoupler quite easily.

Then add and tune a proper compensation network to obtain stability.

Check this guide for an introduction on compensation networks. It is targeted for SMPS power supplies, but the core concepts are the same, that is the need to model the open feedback loop at steady state, introduce a perturbation, and perform an AC analysis to get the frequency and phase response. (The bode plot)

https://www.ti.com/seclit/ml/slup340/slup340.pdf


The LC output filter.

A LC filter is a second order passive filter.
It's goal is to filter the high frequency switching pulses to obtain a pure sine wave with the least amount of harmonics.
In that design, I set the fc (corner frequency) at 10 times the mains frequency, that is 500Hz, not that the lower this frequency, the larger the L and C values, and the larger the effects on voltage with changing load impedannce.

You will notice also that the sine waveform THD will change with loading, appearing less distorted at higher loads

Finally, the LC output stage has an effect on output power factor. An inverter designed for inductive loads will be tuned differently in that regard.

Loading of the filter also affects its response and hence change the output voltage.

Check this thread for a discussion on inverter LC filters :

https://www.edaboard.com/threads/true-sinewave-inverter-output-lc-filter-design-help-needed.375327/

Running the simulation.

Download the zip file in the post introduction. Check the readme.txt in the zip for important information about LTspice setup for that model. Most of the important parameters are parametrized and explained
Most of the inverter functional blocks have commentaries that complement the information given here.
On a core i5 Elitebook 8440p, using LTspice under Wine on Linux, I get around 6.5us/s simulation speed. Try to get at least two or three output sine periods.
This is a work in progress, it will be updated as I improve the model.

A faster approach.

Fortunately, you don't have to reinvent the wheel if you have no time to learn all the intricacies of inverter technology, It is however always better to have an understanding of how all the inverter components work before using a board that implements some of the inverter functions in their design.

For this, there are inverter boards, and we'll have to turn to China and EGMICRO
EGMICRO is a supplier of ASIC inverter boards and more that take care of the SPWM thing, DTI, Gate driving, frequency settings, and voltage feedback, plus they have a monitoring output. They can be used for UPS inverters and also for VFD operation. They also have 3 phase boards.

For single phase designs, there are three ASIC demo boards sold on the market :
- EGS002 is an older but still popular demo board that features the EG8010 ASIC and two IR2110S gate drivers.
I used the same gate drivers in my LTSpice model.
- EGS005 is an newer demo board that features the EG8025 ASIC. It is a monolithic design with integrated gate drivers.

- EGS003 is a board available on the market, with a EG8011 ASIC, plus a single EG2126 package housing a full bridge Gate Driver. But It was since discontinued (the EGMicro website purged the documentation about EG8011) It may have been superseded by EGS005 or it may suffering from issues. Suppliers still sell it. The EG8011 ASIC datasheet still can be found elsewhere. I do not recommend that design if the ASIC supplier made it obsolete.

We will nevertheless perform some intel gathering on these three designs to see how they differ in their reported features.

For three phase designs, There is an EGSO31 board based on the EG8030 ASIC. It uses separate EG3012 gate drivers instead of the more ubiquitous IR2110. A three phase design is necessary for VFD designs powering 3 phase induction AC motors.

To get and idea of how these devices work, you need to check the Demo board, the ASIC and the Gate driver datasheet.
Some of the PDFs are available in simplified Chinese only, but you can pass it to Google translate and have a quite precise idea of what does what after sometimes a little guessing. The quality of Google's technical translations is quite good, since these are terse statements and not litterary works.

The goal is to look precisely at the application schematic and adapt it for our needs.

We will start by investigating the EGS002.

Check that GreatScott's video on EGS002 as an introduction, it also uses a demo UPS board (not just the ASIC+driver board)
 
https://www.youtube.com/watch?v=Dn2PFebi2ww

Please keep tuned for part II. Where we will discuss EGS002 in more detail to unveil its secrets.

230V DC redundant network with a renewable energy source and AC auxiliary power input.

The goal of this design is to get rid of :

  • The inverter stage of UPS or Grid tie inverter technology
  • The rectifier and PFC stage of PSUs

And make use of a renewable power source such as solar panel array

The core of the design is a DC/DC converter with MPPT, current sharing, battery management, It uses an auxiliary AC power input to allow for operation when solar panels do not provide enough power.

Such designs already exist, but they usually contain an inverter stage to output pure sine mains voltage AC.

The hard part of these designs, whether they contain an inverter stage or not at the output is the current sharing stage.
Finely controlled current sharing requires voltage control of all the input sources. However, the output DC/DC step up/down converter stage that conditions solar panel output is dictated by the MPPT algorithm. The solution then is to perform control of the voltage output of the mains AC/DC step up converter stage.

However, by doing so, the load impedance seen by the solar DC converter changes, and upsets the MPPT algorithm that will try to compensate for that change by upseting the DC output voltage.

The current sharing algorithm is non-trivial.

A solution for fast prototyping could use a digital control algorithm to control the voltage of the mains AC/DC converter through a DAC.

A simulink model for such a device is a requirement before attempting any practical device.

It is better if the core UPS also provides battery equalizing/balancing through individual battery/cell links.

Equalizing links to the battery bank are not shown in the schematic.

The UPS has to revert the operation of the converter for battery current draw instead of charge in case of mains failure to supplement solar panel output.

Fortunately, bi-directionnal switch mode converters designs and IC exist for that design. They allow the usage of the same switching transformer for both charge and discharge.

In a 230V DC design, It is also beneficial to have battery banks operating at a voltage close to the operating voltage of other buses, to allow for a switching transformer ratio close to 1:1

The device schematic shows a standard telco 48V battery bank.

The same goes for the solar panel array, it also has the beneficial effect of reducing ohmic losses and requirements for larger cable sections.

Thus the solar DC/DC converter has to be able to accomodate with various solar panel arrays configurations. For this reason a step up/step down design is preferred.

Failure scenarios

To avoid a single point of failure, the core converter is assisted by a standby unit powering up the standby PSU of load devices.

The standby device is not connected to the solar panel array in standard operation and provides power from AC mains utility. It is connected to the battery bank but does not perform charging in nominal operation conditions.

In case of interruption of AC power to the standby unit, it powers itself and provides power to the standby PSU (that draw a very limited amount of power) through the battery bank.

In case of interruption of AC power to both units, the standby unit will perform the same way. The active unit will provide power from the solar panel array with assist from the battery bank.

Switchover operation.

Whenever the active unit encounters a fault, it is reported to the switchover controller, that chooses whether or not to transfer the load to the standby unit by remote controlling the ATS, and also performs transfer of the solar panel array source to the standby unit, if the fault requires it.

ATS should also be able to operate automatically without switchover input (it is an ATS after all) and switch to the standby or active unit in case of DC power loss from either unit.

It is recommended to use a DC/DC SSR based ATS for this operation to reduce switching time and required output capacitance that allow for long hold times of the voltage on the PSU power bus.
High voltage / High capacitance capacitors banks are expensive.

Electromechanical relays also have a limited rated number of cycles before failure compared to SSR. Note that the SSR ATS has to be rated for 230V DC, which can be harder to source than a DPDT relay for the same ratings.

A custom MOSFET based design can be implemented with either low side or high side switching or both. low side switching is prefered because of lower RDSon for NMOS devices, or the use of a voltage pump for high-side NMOS switching. This design implement both high side and low side switching to completely isolate the the active and standby core DC UPS.

Failure of the ATS to switch to the standby unit will be compensated by the standby DC PSU providing the load to the device, standby DC PSU bypass the ATS in this design.

Note that a non recoverable failure will happen in case of switchback failure of the ATS to the primary unit, if the secondary unit fails to provide power to the load through the standby PSUs

ATS operation would sense voltage on the input buses, and initiate switchover / switchback if the DC voltage falls under a specified threshold, above the UVLO threshold of the load PSUs

The switchover controller should also be operated from a separate DC source with battery backup.

Earthing considerations

Note that the core converter units have their chassis connected to the main earthing bar, through the AC power cable since they have an utility AC input.

This design shows a TT earthing arrangement, but can be adapted to other earthing schemes.

DC outputs are isolated, and the whole DC bus is floating. Devices chassis and racks should be bound to earth.

Protective devices

Adequate DC overcurrent protection devices should be present on the DC bus before and after the ATS.

Battery banks should be protected by adequate fuses

AC mains supply employs SPD, RCD and OCD

Final note

Such a design is hindered by the relative novelty of monolithic MPPT/Auxilliary mains AC input/battery chargers without an inverter stage, and also the lack of 230V DC PSUs in the market.

Most DC PSU in the market are based on the old telco standard of using 48V, which is non-optimal for ohmic loss reduction, and require larger section cables.

The 48V voltage is also not on a comparable level to the high voltage outputs of modern solar panel designs, that connect panel modules in series for optimal and less costly power transmission from the array to the MPPT unit.

DC overcurrent protection is also more costly than the AC counterparts.

We believe however that this field will show new technology advancements in the following years.

Linux to Windows backup using Cygwin and Rsync

This procedure may be helpful for backups of Linux VMs on the cloud. Some (most) service providers make it notoriously difficult to export VM images or snapshots out of the cloud.

While it is still possible to perform an image backup using Clonezilla, again, not all service providers allow boot on an ISO for a given VM. In these cases, a helper volume should be added to the VM and have Clonezilla installed on it, and selected as the boot device. The original boot volume would then be backed-up to an image locally on the helper volume and exported out of the cloud after the process completes. Although that is the theory, I have not tested this specific process myself. Keep in mind that some providers VM instances only support one volume attached. Also, this process is a one-shot operation, and is hardly automated, and creates downtime.

There are other block level device time consistent backup strategies you can try though if you use LVM and have space on the LVM group, it is possible to snapshot your volume, mount the snapshot, and have rsync transfer it over the network. Keep in mind that if the rsync process is initiated on the backup system rather than on the backed-up system, you’ll better have to use ssh or other remote command tools to perform snapshot creation and mount and then rsync, have a way to check for errors returned by these commands and then only initate rsync on the backup system. Then perform a post-backup remote execution to unmount the snapshot and destroy the snapshot. This whole process is a bit similar to the Windows VSS snapshot operation.

For more information about this process, check :

LVM snapshot Backup and restore on linux

If you don’t use LVM then you’ll have to resort to backups that are not time consistent (not frozen in time), but it is better than nothing. That is the object of this tutorial.

Setting up cygwin on Windows.

Download cygsetup and perform a full or a minimal install. For a minimal install, you’ll need SSH and Rsync as well as it’s dependencies.

Now you’ll have to choose between using rsync through a SSH encrypted channel, or through a rsync daemon running on the remote host to be backed up.

Using rsync through SSH for a full system backup requires a root login through SSH, and automating rsync with SSH will require a way to supply credentials automatically, unless using a public/private key non interactive (passphrase less) authentication scheme.

In the case of SSH plain password authentication, supplying it to rsync can be done through sshpass, which exists as a Cygwin package, but I have not tested it myself in conjunction with rsync

http://www.cygwin.com/packages/summary/sshpass.html

https://stackoverflow.com/questions/3299951/how-to-pass-password-automatically-for-rsync-ssh-command

However, allowing SSH password root authentication plus storing its password as cleartext in a file for sshpass to use it is a huge security risk.

At least, with a passphrase less public/private key pair, the problem amounts to securing the private key well on the filesystem. It will still be acessible (read only) in the user account context that runs the rsync/ssh script on Windows.

For all these reasons, I find it preferable to use the rsync daemon on the remote system. Which allows to use a backup account instead of root.

The downsides however are that you need to open the rsync TCP port on the remote system and configure your firewalls in the connection path accordingly; and also rsync daemon does not encrypt traffic. If it is an issue, then use a VPN or an IpSec tunnel.

Setting up the rsync daemon on linux Debian

apt-get install rsync

Edit the rsync daemon configuration file.

vi /etc/rsyncd.conf

Here is a sample of the rsyncd.conf

uid=root
gid=root

[share_system_backup]
	path = /
	comment = system root
	read only = true
	auth users = backup
	secrets file = /etc/rsyncd.secrets
	hosts allow = <ip_of_backup_system>

As per the rsync manpage, rsyncd runs as root, and uid and gid parameters, which can be global to all rsync shares or share specific, specify the impersonated context of the rsync access to the filesystem.

Since we’ll perform a system wide backup, we’ll use root.

auth users specify the rsync users authorized to access the share. These users are rsync specific, not system users.

read only is used since no writes will be performed on the backed up system, unless you want rsync to upload some state file on the remote system after backup, as we won’t be using SSH, that is a trick way to signal something to the backed up system, without any remote execution.

hosts_allow is useful for a cloud VM that does not have firewalling options provided by the service provider.

The user login password pair is specified in /etc/rsyncd.secrets.

vi /etc/rsyncd.secrets

backup:346qsfsgSAzet

Use a strong password.

Next start the rsync daemon, check it runs and check its socket is listening on TCP port 873.

rsync --daemon

ps-auwx | grep rsync && netstat -nlp | grep rsync

Then we’ll make rsync launch at system boot.

vi /etc/default/rsync

Change RSYNC_ENABLE to true to start the daemon at system startup through init.d

Rsync configuration on Windows

We’ll start by setting up the windows batch files that will prepare the cygwin environment and run rsync. Change <IP> with your remote system to be backed up IP or DNS name. This script assumes standard port 873

The default file is named CWRSYNC.CMD and should reside at the root of the cygwin64 base folder.

@ECHO OFF
REM *****************************************************************
REM
REM CWRSYNC.CMD - Batch file template to start your rsync command (s).
REM
REM *****************************************************************

REM Make environment variable changes local to this batch file
SETLOCAL

REM Specify where to find rsync and related files
REM Default value is the directory of this batch file
SET CWRSYNCHOME=%~dp0

REM Create a home directory for .ssh 
IF NOT EXIST %CWRSYNCHOME%\home\%USERNAME%\.ssh MKDIR %CWRSYNCHOME%\home\%USERNAME%\.ssh

REM Make cwRsync home as a part of system PATH to find required DLLs
SET CWOLDPATH=%PATH%
SET PATH=%CWRSYNCHOME%\bin;%PATH%

REM Windows paths may contain a colon (:) as a part of drive designation and 
REM backslashes (example c:\, g:\). However, in rsync syntax, a colon in a 
REM path means searching for a remote host. Solution: use absolute path 'a la unix', 
REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the 
REM drive letter:
REM 
REM Example : C:\WORK\* --> /cygdrive/c/work/*
REM 
REM Example 1 - rsync recursively to a unix server with an openssh server :
REM
REM       rsync -r /cygdrive/c/work/ remotehost:/home/user/work/
REM
REM Example 2 - Local rsync recursively 
REM
REM       rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/
REM
REM Example 3 - rsync to an rsync server recursively :
REM    (Double colons?? YES!!)
REM
REM       rsync -r /cygdrive/c/doc/ remotehost::module/doc
REM
REM Rsync is a very powerful tool. Please look at documentation for other options. 
REM

REM ** CUSTOMIZE ** Enter your rsync command(s) here

echo "start" >> c:\scripts\rsync_sys.log
date /t >> c:\scripts\rsync_sys.log
time /t >> c:\scripts\rsync_sys.log

rsync --no-perms --itemize-changes --password-file=rsync_p -lrvcD --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/root/*"} backup@<IP>::share_system_backup/ /cygdrive/d/BACKUPS_FROM_CLOUD/SYSTEM >> c:\scripts\rsync_sys.log

date /t >> c:\scripts\rsync_sys.log
time /t >> c:\scripts\rsync_sys.log
echo "stop" >> c:\scripts\rsync_sys.log

You’ll need to add a file named rsync_p in my example that contains the password specified for the backup user, matching the password defined on the rsync daemon host. This allows non interactive execution of rsync.

Place this file at the cygwin64 base folder level.

Secure that file well at the NTFS level.

itemize-changes will log in the rsync_sys.log the file operations performed on the system. It is useful for troubleshooting.

http://www.staroceans.org/e-book/understanding-the-output-of-rsync-itemize-changes.html

The -lrvc flags tells rsync to copy symbolic link information (absolute and relative). You can test that is the case by using ls -ltra through a cygwin shell, recurse directories, be verbose, and check for changes based on checksum instead of file timestamps, which could be useful if there are timestamp discrepancies because of the cygwin environment. It is worth testing it’s effect with a test run.

Also, I excluded the standard list of directories recommended to be excluded from a system backup for Debian systems using the –exclude option.

Permissions / ACL issues

Note that I use –no-perms for rsync. That is one caveat with rsync running on cygwin, since it uses POSIX to Windows ACL translation, copying perms COULD render the files unreadable by other processes on the Windows system if permissions from the source system are preserved, On the other hand, preserving source permissions could make baremetal recovery possible. It is worth experimenting with this option.

A way to circumvent this problem when using –no-perms is to back up permissions separately on the linux system using getfacl and setfacl for restore operations. keep in mind that getfacl ouput is substantial in terms of size and getfacl does not support directory exclusions. It may involve a bit of scripting.

Example of getfacl script (backupacl.sh)

cd /home/backup
getfacl -RL /bin > filesystem_acls_bin.dat
getfacl -RL /boot > filesystem_acls_boot.dat
getfacl -RL /etc > filesystem_acls_etc.dat
getfacl -RL /home > filesystem_acls_home.dat
getfacl -RL /lib > filesystem_acls_lib.dat
getfacl -RL /lib64 > filesystem_acls_lib64.dat
getfacl -RL /media > filesystem_acls_media.dat
getfacl -RL /mnt > filesystem_acls_mnt.dat
getfacl -RL /opt > filesystem_acls_opt.dat
getfacl -L /root > filesystem_acls_root.dat
getfacl -RL /sbin > filesystem_acls_sbin.dat
getfacl -RL /snap > filesystem_acls_snap.dat
getfacl -RL /srv > filesystem_acls_srv.dat
getfacl -RL /tmp > filesystem_acls_tmp.dat
getfacl -RL /usr > filesystem_acls_usr.dat
getfacl -RL /var > filesystem_acls_var.dat

tar -czvf backup_acls.tar.gz filesystem_acls_*
rm -f filesystem_acls_*

You can use xargs and a $ placeholder plus a list of folders to backup permissions from to make this script a one-liner.

the backup_acls.tar.gz will be backed up by rsync when it runs.

In this example, ACL backups and the backup script are stored in /home/backup. As said before, backup is not necessarily a linux user since rsync maintains its own authentication database, but I use this directory for convenience.

In this script backup of /root ACLs is not recursive as a dirty way to remove the .wine directory that contains symbolic links to the whole file system, and thus would make getfacl to follow them and spend a huge useless time (and filespace) for directories such as /dev, /proc /sys. So it’s possible for you to add the recurse flag if this problem does not affect your system.

Run the batch for tests and schedule it by adding it to /etc/cron.daily/ for instance. It should run in the root context.

A note about restores and symbolic links

the rsync -l options copies symbolic links, which means thar you have to use rsync too for restore operations, Don’t expect symbolic links to magically reappear if you use WinSCP or another tool to restore a folder.

More on this subject :

https://www.baeldung.com/linux/rsync-copy-symlinks

Testing the backup.

Edit the cwrsync.cmd to add the dry-run option, run the cwrsync.cmd through a cmd shell and examine the rsync_sys.log file, If everything seems OK, then you can remove the dry-run option, and run it again. The first run will take a long time, use the start and stop markers in the logfile to check how long it takes once finished. rsync will tell you the speedup due to the delta algorithm, it should increase on subsequent runs.

Scheduling the backup.

Use Windows task scheduler to run the cwrsync.cmd script. In my case, I use a privileged account. It is advised to test with the least amount of privileges for cygwin to be happy and for filesystem operations to succeed.

It is advised the turn on scheduled tasks history. And very important, make sure that the “start in” directory for the cwrsync.cmd action is set to the base cygwin64 folder.

Test the backup run using a “run once” trigger at t+1 minute in addition to the recurrent trigger.

A final note.

This backup strategy will ensure that you have an updated state of your remote system on the Windows system. It is similar to an incremental backup that patches the destination directory. It does not store any history of files besides the last backup operation.

I don’t use the –delete option that propagates deletions of files and folders from the source to the destination, has it can be very impactful for a disaster recovery use where there is no file versioning history. Note however, that this will bloat the backed up structure on the destination and make recovery efforts more complicated in the long term.

The best option for a backup if you use LVM, albeit using more space on the destination is to use cygwin + rsync to backup a tar gz file of a LVM snapshot directory structure instead of the directory tree of “/”. If you go this way though, it is advised, as already stated above, to perform the snapshot operations synchronously from the backup system through the cwrsync.cmd using SSH, before the rsync operations. That probably means using a passphrase less public/private key pair for ssh to run these commands interactively. Don’t forget to unmount and remove the snapshot after rsync is done.

Design and fast prototyping of a milk curd stirrer and heater using Arduino.

Foreword.

This project adhere to the DIY/maker ethos and use a fast prototyping, modular electronics approach.
It also demonstrates that a project can be done even without all the ‘proper’ tools, situations that are encountered
every day in workshops of less developed and fortunate countries, or simply in bad luck situations or when working in a secondary workshop that is not well furnished.
Some tools are non negotiable, but at other times, all it requires is a little inventivity, a bit more skill, patience and the zen spirit.

This is my first true project involving mechanics and electronics. I consider myself with a bit of experience under the belt in electronics, more experience in MCU coding, although
my code is crap before I decide to clean it up and properly factorize (which takes a long time to be put in motion) specially when i code in a rush (that was the case in that project).
I’m sorry for that.
As for mechanical engineering, I am an absolute beginner.
For the thermodynamics aspect, a little research online helped quite a lot.

There are overall a lot of lessons learned and practices confirmed to be important in completing a somewhat complex DIY project.
The more important are, in my case were :

For the idea to proof of concept part, check for state of the art using online resources, specifically video advertisements of similar devices, and see what features you can keep
from them. also check other makers work, and apply the Japanese philosophy of making an already existing product better if you dare.
careful beforehand planning of operation steps. write things.
Ordering of the operations so they don’t conflict each other and make another step impossible because of oversights is the most difficult part, at least for me.
CAD is advised to have a virtual look at the project and is good for post project documentation, although it takes time, and won’t be of real help
if you don’t use machining shops in your project.
May be useful in the future though if you decide to produce more, and also as an advertising and proof of seriousness in your endeavor.
Document your project during execution of the machining, packaging, and all the ‘practical’ work. it will serve you later by giving you a precise
view of the challenges you encountered, so that you can refine the production cycle through continuous improvement. sometimes a picture may be enough.
Video is paramount if you want to make your project public for other DIYers, and advertise it. It is hard to find the time to take pictures or videos when you’re in
the flow doing shop work, because it may break that flow. take it as a break time. making the video useful by being always centered on the work area is the big time cruncher.
In these cases, pictures are a perfect compromise.
Take inspiration of the hackaday like sites to see how experienced makers and hackers document their project.

analyze f***-ups in detail when they happen (root cause analysis, including psychological factors that lead to the mishap) so they don’t happen again
assumptions are the mother of a lot of them.

a great part of the skill required comes from an intuitive feel of Newtonian physics in the conception phase, and dexterity in the shop phase.
Look at a lot of DIY and shop videos to learn skills you don’t have beforehand.

compile the CAD files for parts and from the electronics EDA tool so that your BOM is comprehensive, and order more than required for cheap parts, especially if sourced
from China. sourcing skills are also something you’ll need to work on, and also it is quite fun to browse marketplaces when it is not for mindless consumerism.

Good luck to you !

1) Presentation.

milk curd stirrers are one of the critical devices used in the dairy and cheese industry.
A professional device usually performs these operations :

  • Precise control of heating (heating slope and temperature hold control, multi-step heating)
  • Stirring
  • Curd cutting using a cutting grate.

Some devices implement passive or active cooling using a refrigerant coil.

Professional devices are usually large, several hundred litres devices in the entry level market.
This scale is not suitable for small family farm operations.

This device performs heating control using the PID method (proportional / integral / derivative) and control stirring (on/off).
Curd cutting has to be performed by hand but there are several methods envisioned to expand the prototype into making this process automated.

Vessel used is a 15L Gredil brand stainless steel pan.

1.1 Heating control.

Using a standard 1200W kitchen hotplate that has been wired to be controlled by an SSR relay (fotek brand) and a PWM signal, heating duty cycle can be defined by the
PID algorithm.

  • Heating phase uses open loop control with inputted estimations of thermodynamic parameters (vessel geometry, external temperature, quantity of milk, heat capacity & transfer terms)
  • Heat transfer terms take into account conductive and convective effects.

Due to the large mass of the heating plate, thermal inertia is quite high and has to be compensated with a large derivative parameter in the PID controller loop.
Ceramic is a good choice, induction plates even better, although there are few single plate portable models for these technologies. Price tag is higher.

Closed loop control kicks-in at a defined set point before reaching temperature hold plateau to minimize temperature overshoot.

A thermoter probe well is present and secured to the base plate to perform temperature measurement as close as possible to the project center of mass of the liquid to be heated.

1.2 Stirring.

A 28W low rpm single phase AC motor, geared for 15 rpm, provides enough torque to enable consistent stirring. The motor is coupled to a 8mm shaft, ending with two perforated 12*15 cm , 2mm wide 308 stainless steel plates, with each having 5 33mm diameter holes. the shaft is terminated on its contact point on the steel pan with a conical taper to reduce friction between the two. The shaft is secured by a standard 8mm flanged bearing to the base plate crossbar.

Not included in this prototype, but highly recommended is to provide a rotor lock detection mechanism by monitoring the current drawn by the motor, and disabling it in case of overcurrent. Monitoring the motor temperature is also recommended based on the provided datasheet parameters.

1.3 Protection and conformity devices.

The devices uses two DIN circuit breaker inside a 2 breaker box. one 10A for the hot plate power, one 2A for motor and Arduino power. the circuit breaker box is secured by screws to mounting struts across the two slotted aluminum profiles, earthing of the whole metallic chassis of the device is provided at this point.
Note that fast period PWM induces switching noise on the mains AC. An adequate line filter for a 1200W hot plate is recommended.

1.4 The controller.

An Arduino Mega 2560 with an LCD 1602 keypad shield is used to perform the following operations :

-Motor on-off control for stirring and hot plate PWM control
-Temperature measurement
-Liquid quantity measurement using a laser distance measurement module.
-User interface (LCD 1602 keypad shield) to input program parameters and program selection

Liquid quantity measurement is tricky and one need to carefully plan the placement of the module based on the datasheet recommendations.
Specifically, the hole through through the baseplate and electronics enclosure has to be wide enough not to disturb the laser cone, and it is also very important that the PMMA protection window is mounted as close as possible to the module, ideally in contact with it. A distance larger than a couple of mm is sufficient to induce large crosstalk, making the measurements lose precision. at 10mm distance, measure is no more effective. The placement of the module on the horizontal plane also has to be made distant enough to the vessel walls for the same reasons. Two more constraints : the protection window has to be thin and provide adequate IR transmittance. PMMA is a good choice. Also, water vapor condensation is a problem for hot running measurements. A hydrophobic treatment of the PMMA window is therefore required.
On the other hand, milk is quite reflective and allows rather precise measurement to a 5% to 10% error margin.
This module requires quite a lengthy process of calibration : distance offset, measurement of materials with different reflectance, and crosstalk calibration.

A simpler approach would simply to ask the user to input the quantity of milk present inside the vessel.

2. Bill of Materials.

  • Aluminum strip 30mm x 2mm (for struts between the aluminum profiles, and to secure the breaker box) 50 cm
  • slotted 20×20 aluminum profile, length depending on the vessel depth, and adaptability for varying depth vessels qty 1
  • 68KTYZ motor, 10 rpm qty 1
  • 68KTYZ right angle mounting bracket qty 1
  • 8mm 303 stainless steel shaft, length depending on the vessel depth, and possibility of using vessels of varying depths qty 1
  • 8mm ball bearing with mounting holes, model KFL08 qty 1
  • aluminum shaft D19L25 coupling nut 7mm to 8mm qty 1
  • stainless steel 309 plate, 12x30cm 2mm width. This depends on the vessel geometry. used for both paddle wings qty 1
  • slotted profile 20×20 nuts, to secure struts, motor bracket, and plate to profile mounting brackets qty 12
  • aluminum profile 20×20 90° corner brackets, to secure aluminum profile to baseplate qty 4
  • ABS electronics enclosure 140x110x35mm Gainta Brand G738 qty 1
  • thermometer probe on cable DS18B20 qty 1
  • thermometer well, 20 cm length, 11mm width qty 1 2 DIN breaker wall mounting box (PawBol C.2030) qty 1 Arduino mega 2560 qty 1
  • perfboard (at least 140×110 mm) qty 1
  • LCD 1602 Arduino keypad shield qty 1
  • laser time of flight (distance measurement) module TOF050C (VL6180X based) qty 1
  • standard 5V relay module qty 1
  • Fotek SSR40DA solid state relay (5V DC control) qty 1
  • Hi-Link 5V power supply module, 3W qty 1
  • 275V varistor qty 1
  • 4.7K resistor (for temperature probe) qty 1
  • Silicon wire hookup cable, 22AWG qty 1
  • M2,M4 and M5 screws qty 1
  • IEC C14 male cable connectors qty 2
  • IEC C14 female connector qty 1
  • schuko cable C14 female terminated qty 1
  • 1.5mm2 conductors, blue, brown and green/yellow, 1m
  • cable relief and clamp nuts (for cables entering the electronics enclosure) qty 3
  • 1200W hotplate qty 1
  • 15L stainless steel vat, Gredil brand qty 1
  • U bolts. geometry will depend on the stainless steel vat handles position qty 2
  • butterfly bolts to secure U bolts to the baseplate qty 4

3. Tools needed

  • bench drill and HSS metal drills up to 12mm diameter
  • angle grinder with cutting disk and grinding disks
  • drilling and cutting oil
  • threading taps
  • 33mm hole saw for metal
  • stainless steel 2mm stick welding electrodes
  • stick welding inverter
  • perforated welding table or at least two vices, with one vice with XY axis regulation
  • spacers of varying widths to assist in vertical position of the paddle during welding
  • piece of copper to use as heat sink for stainless steel welding (to prevent warping)
  • a dremel with 28mm cutting disks and small abrasive (diamond) milling bits with various forms
  • soldering iron
  • multimeter
  • power screwdriver and angle grinder (to make the conical taper at the end of the shaft) or better, a lathe.
  • caliper and measuring tools
  • sharpie pen

4 Making the prototype

4.1 preparation of the baseplate

clean it and remove any protective film. mark the center with a sharpie. this will be the hole for the shaft.
drill the shaft hole some 20% larger so that there is leeway in final positioning of the shaft and bearing assembly

4.2 mounting the aluminum 20×20 profile

cut your aluminum profile in two parts (if applicable). the original extremities will be positioned over the baseplate,
since it is not certain that your cut angle is perfectly perpendicular if done with an angle grinder or a hand saw.

4.3 mounting the motor 90° bracket

place all T slot nuts inside the profile and position them for screwing the bracket. ensure that the base of the motor bracket is
parallel to the baseplate during test positioning. Screw the 68KTYZ motor to the bracket, install the shaft coupler, pass the shaft through
the bearing, position once again the assembly to the baseplate passing the shaft through the baseplate hole. test for shaft perpendicularity
using a protractor and you eyeballing abilities, using a protractor oriented in X then Y axis. beware of parallax. Note that this is not critical
since shaft coupling nuts have a helical groove to allow precession of the shaft, but it’s better to have the habit to do the right thing.

mark with a sharpie the outline of the corner brackets of the profile/motor assembly and the center of the holes of the brackets and shaft.
Check that the shaft does not contact the baseplate hole.
drill the holes for the baseplate to profile L brackets, again a bit larger so there is leeway for final positioning and screwing. once drilled secure the baseplate to the profile/motor assembly while checking that the assembly sits so that the shaft is perpendicular to the baseplate, again use protractor or a fancy laser. also at this point position the baseplate so the
shaft comes into contact at the center of the vessel bottom.

4.4 U-bolt baseplate to vessel fasteners

At this point it’s time to prepare the u-bolt holes that will be used to secure the whole device to the vessel using its handles.
this part is critical.

mark the contact point of the vessel lid to the baseplates, 4 points in total. will be useful if it moves inadvertently.

Since there is no contact except at the lid of the vessel, you’ll need to use a square ruler with a large contact area to project the handle of the vessel to the baseplate, offset to the centerline (axial) of the handle cylinder, and add offset of half the distance center to center of the u-bolt, mark the holes and drill. better drill to exact u-bolt bolt body dimensions, so there is no room for it wiggle if the butterfly bolts come loose during operation.

once again unscrew the motor/profile assembly and remove the shaft, and drill the u-bolt holes.
after that repeat the positioning steps above with the motor/profile assembly on the baseplate secured, and the baseplate secured to the vessel using u-bolts and check that
the shaft once again contacts the center of the vessel.

4.5 layout of thermometer well and electronics enclosure

test layout of the remaining components all at the same time :
the thermometer well, the bearing orientation, and the electronics enclosure.

there is a constraint on the VL6180X laser distance sensor (the laser cone of avoidance). some trigonometry will be helpful to define the minimum diameter of the electronics
enclosure window and the baseplate window. verify that the cone divergence does not intersect the wall of the vessel.
you can also skip this part because it is quite tricky and go for static liquid quantity input by the user.

4.6 securing the ball bearing to the baseplate

mark the holes for the ball bearing while the motor/profile assembly is correctly positioned.
remove the assembly again
secure the ball bearing. position and secure the motor profile assembly again, check that the shaft does not contact the baseplate.

4.7 thermometer well installation.


you can either place the thermometer well in the hole, it will rest on its flange, (quick and dirty) or glue a NFT threaded adapter to the baseplate so you can screw in the well and it does not move. you’ll have to carve out a slot in the paddles so that the well does not collide with the rotating paddles

4.8 define the layout for the components inside the electronics enclosure

at this point you should have the positioning constraint for the distance laser module on the XY axis.
non soldered components are the SSR and motor relay : they will be freestanding inside the enclosure and secured by the perfboard on their sides
the Arduino and LCD shield assembly will be secured to the top part of the enclosure.

Drill two holes in the perfboard at least, diagonally, you’ll use the same screws to secure the perfboard to the enclosure and to the baseplate.

position and mark baseplate with a sharpie through the bottom cover holes.
Drill the baseplate holes. baseplate holes maybe threaded or not and you’ll use bolts. it’s up to you. I tapped threads and was lucky. beware that threads in aluminum are soft compared to steel. threading is less robust but a little more elegant than bolting.
test mount bottom cover and perfboard to the baseplate.

Carve out the perfboard for SSR, relay and Arduino footprints. This step is required if you use the enclosure specified (Gainta) since there is not enough space on the Z axis for these components. if you find a better enclosure on the market, lucky you ! or just 3D print one.

The good side of the tightness of placing was that the perfboard maintained the two relays in place. better carve out less so that you finish with a
mill bit on the dremel and obtain a tight grip of the perfboard to the components.

also note that I had to chop parts of the fotek SSR plastic near the connecting terminals to connect the washer type electrical connectors.

of course, before carving out, you have to make sure that the remaining components fit on the little perfboard you have left : that is, the varistor on the AC side, power module, and the thermometer module resistors, maybe an electrolytic capacitor on the PSU output.

4.9 remaining holes on the bottom part of the enclosure

the bottom part of the enclosure will have three additional holes for the wire ports, two at the side closer to the center of the device :
one for the hot plate electrical wiring input to the enclosure (phase, neutral, and ground) the other port for the stirring motor power output (AC), and the MCU + motor power input (AC)
the distal hole from the center is for power output going to the plate.

If you have a precision bench drill use it for the holes in the enclosure, and finish with a dremel when required. the port holes are better milled with the dremel, since there is usually no
clearance to drill them with a precision bench drill, unless your setup allow to do so.

so, to sum up, the enclosure bottom part has these holes :
2 mounting holes to secure to baseplate and perfboard
3 holes for cable routing

finally, one of the enclosure plastic insert sides should have a hole for the arduino USB connector.

4.10 Soldering the components to the perfboard

First, A note about the VL6180X module

if you decide to implement it, remember that the PMMA window has to :

  • make the enclosure airtight and watertight
  • be in contact with the module itself to lower crosstalk to a minimum
  • be fog treated
  • be heat resistant. better measure the temperature at that point at the maximum operating temperature and liquid quantity of the device. 92°C liquid temperature is used for milk sterilization

that means that the Z axis section for the laser module has to be layered as this (going downwards) :

  • perfboard
  • laser module (using 90° header pins)
  • pmma window
  • enclosure bottom cover
  • baseplate

the PMMA window should be way larger than the hole so that the epoxy gluing step does not smear glue down the hole. use the least amount of glue over a rectangular path,
so the enclosure stays airtight.

Remember step 5 and check underside that the VL6180X chip is well inside its positioning margin so that the laser cone does not interfere with any part. re-mill if needed.
better perform calibration before gluing the window. the process is outside of the scope but there are well documented in the datasheet. You’ll have to write into quite a lot of registers.
on the VL6180X, check my code (will comment it better soon)

Also the module will be soldered on the perfboard bottom layer. a dual layer perfboard is thus required.
Finally, the perfboard may not be in contact with the bottom of the enclosure because of the module, in the case you do not use the provided mounting holes on the enclosure, as they provide some clearance from the bottom cover. if I remember well, I could not use them because of the crowded layout. This may lead to the bending of the perfboard, and axial force on the perfboard/cover/baseplate screws. if you manage to add plastic standoffs washers over the screws between the perfboard and the bottom cover, it will give a better result.

4.11 wire soldering to the perfboard (perfboard to perfboard and perfboard to freestanding)

pass the DS18B20 wires inside the wire restrain port and solder them to the perfboard.
solder the AC carrying wires leads (low amperage, these will connect to the 2A breaker and supply power to the module and motor) to the varistor and pass them through the port, later they will be bundled with the hotplate carrying wires. this could lead however to stronger switching noise coupling from the PWM operation to the AC power lines to the controller and motor.
In my case I did not use any line filter in the enclosure since there was no room left, but the hi-link module probably handled this well as controller operation is stable.

solder the hi-link power supply, and all interconnect wires.

this project feeds 5V to the Arduino through the 5V pin, so it bypasses the regulator, thus an electrolytic capacitor is advised to smooth DC when the motor relays kick in.
hi-link power supplies are good if you get the real thing, mine had to be opened, and a soldered wire was loose so it didn ot supply 5V. either i did not have luck or it was
counterfeit crap. lucky that it was just a wire issue. Always test ‘complex’ modules and components on breadboard first ! and if you can, order more than required since they’re cheap
and take time to get to you. Always think about logistics in any project !

One more advice :
I made the rookie mistake to use single stranded breadboard wires. don’t do that, they are fragile as f*** and break on torsion. used proper multistranded hookup wire.

label all the hookup wires on the perfboard side with a Dymo.
secure for the final time (hopefully) the perfboard, bottom cover, and baseplate together.
put the Fotek SSR and the relay in their designated perfboard placeholders
additional internal wiring using DuPont connectors are 5V,GND,relay logic for the motor relay, SSR relay logic and SSR DC ground.
the Chinese 5V relay modules provide male header pins. I advise not to directly solder hookup wires since that will make any servicing difficult.
Fotek SSR use large screw terminals the like used on overcurrent protection breakers, you’ll have to use washer type electrical connectors.

4.12 hot plate power power wiring

Pull neutral and ground 1.5mm2 wire through the enclosure through one of the ports on one side, across the enclosure, to the exit port.
split the same length of brown phase wire in two and solder a washer type electrical connector on one end of each wire, connect them to the AC part of the Fotek SSR.
the connectors will connect one one side to the dual pole 10A breaker on one side, and to the female (important that you use Schuko IEC C14 female connector here to prevent electrocution from the potentially energized prongs when it is disconnected from the plate) Schuko connector that will connect to the male ended Schuko connector cable going to the hot plate.

4.13 prepare the Arduino and the LCD 1602 shield

unfortunately the LCD 1602 shield pin headers come into contact with the top cover in my configuration. the easiest way for me was to saw them off with the dremel.
take care to not let drop the shavings mindlessly.

next solder hookup wires to the LCD shield for :

  • DS18B20 data
  • SSR relay logic
  • motor relay logic
  • 5V
  • GND

And if you use the I2C VL6180X laser distance module, add these cable leads (that’s because the LCD1602 keypad does not route the SDA/SCL through its pins, what a shame) :

  • SDA
  • SCL

4.14 prepare the top part of the enclosure

The fact that the LCD keypad shield plus Arduino have to be secured to the top part of the cover introduces a severe constraint for wiring the whole project.
That’s all Aliexpress had to offer in 2022.
This whole part will have to be rethought in any next iteration, with ideally an LCD keypad solution with ribbon cables and board to board JST connectors, and a PCB plus embedded Arduino on the bottom part, and a larger enclosure on the Z axis.

The top part will have a carved out rectangle for the LCD display, 4 holes to secure the Arduino/LCD and 5 holes for the keypad buttons,
Mill an opening on the enclosure top cover for the LCD1602 display and the five buttons keypad, with the dremel using cutting disks for the opening and milling bits for the keypad. you’ll also need mounting holes, beware when milling there is very little clearance between the holes and the display opening. go slow.

finally connect the shield to the Arduino and mount the Arduino with the shield to the top cover. screws and nuts required.

4.15 connecting the hookup wires leads from the Arduino to the wire leads going to the perfboard, the SSR, and the relay.

put a heat shrink tubing section on the cable solder all hookup wires from the perfboard / relays to the corresponding hookup wires of the Arduino/LCD, place the heat shrink tube over the solder and apply heat.
Of course, for this part you can go for ease of maintenance and just use DuPont male to female connectors cables to connect the bottom cover components to the Arduino, I just trust them less in the long term.
This part is a bit cumbersome since it has to be done with the enclosure partially closed, just to expose the wires.

4.16 closing the enclosure

once everything is in place close the enclosure slowly, checking not to pinch any cable, and screw and tighten the top cover.

4.17 circuit breaker box installation.

While one side of the parallel two 20×20 profiles is used for the motor and its bracket,
the other side will be populated by the electrical breaker box. I used a model for two DIN breakers, since I’m in Europe.
the box will be mounted on the two aluminum strips that serve also as struts. I made two diagonal holes to secure the box, one of these holes will also be used to ground
the device chassis, by the means of a washer electrical connector. grounding for the hot plate has to be provided, so you will need a 3P Wago to extend the earth or use two washer
connectors on the grounding point.

functional tests of the completed device will be covered in another post.

4.18 shaft conical taper

use a lathe or if you haven’t any, make the shaft spin on a power screwdriver, with support along the axis with a ball bearing secured on a vice.
have someone assist you or clamp the power driver trigger so it spins without assistance. once the shaft spins, make the conical taper using the angle grinder.

4.19 paddle cutting and perforation

using a metal hole saw, drill the holes on each paddle, using the same layout for both paddles. check the CAD model.
use plenty of drilling oil and stop drilling and let the saw cool down if smoke appears.

4.20 paddle welding

This part is tricky, because the fixture is a bit complicated to make when you don’t have the proper tools. Best is to have a true welding table with different types of clamps. If you have no welding experience, better give the work to a welding shop.

Without adequate tools you can make a somewhat ok-ish fixture using one vice to secure the shaft, and another 2 axis vice. you will need some spacers placed under the vice to adjust the them on the vertical axis relative to one another, so that the paddle width centerline is aligned diametrically to the shaft.
Then adjust XY axis on the vice that hold the paddle, just above the conical taper. better to have some margin if the paddle ends misplaced so that it does not contact
the vessel when you’ll test it, it will spare you some grinding. in my case, i ended with a max. 5mm margin between the paddles and the vessel, on the base and on the side.

If you’re a beginner like me, it should work (not ruining the parts) but aesthetically it could be crap.
I made a 45° bevel on both sides of the edge to be welded to create a groove to improve the weld robustness.
not sure if it really helped though, better ask a professional.
I used 2.8mm 308 stainless steel rods at 80A, I think it is a bit too hot, but I don’t know if my crap welder is calibrated for amp measurement.
The most important thing is a precise alignment of the paddle to the shaft and to use one or two copper heat sinks contacting both the paddle and the shaft close to the
weld area, underside of course. even like that, the shaft may warp. do not be too alarmed if it does though, because when welding the reverse side, the warping will revert, but because of this you could have trouble to make precise contact to the copper heat sink though.
repeat for the other paddle.

I learned that stainless steel welds may need passivation or electropolishing after welding to be fully corrosion resistant. there are DIY electropolishing methods involving strong acids like phosphoric acid and current, passed through some mineral fiber soaked in acid, and connected to an electrode. check Youtube.

4.21 final paddle assembly rotation tests

make free wheel shaft/paddles spinning tests with the motor coupling Allen nuts loose. it should not contact the vessel sides or bottom. A final test will be required with the motor on.

upload a test firmware to the Arduino and test the relays. at this point the motor relay is the focus of our attention, make the paddle spin, and if it contacts the vessel, grind the paddles at the point of contact until they no longer do.

4.22 breaker box to enclosure wiring connection, enclosure to motor wiring connection, enclosure to hotplate wiring connection, breaker box to main AC input connection

To sum up, there are three bundle of cables : phase, neutral, protective earth for the hot plate, passing through the enclosure and going to the 10A breaker, plus phase and neutral going to the 2A breaker. (since there is no line filter the protective earth does not need to be routed to the controller enclosure) these will provide power to the controller and to the motor through the relay.

and finally a bundle of cables exiting the other side is going from the enclosure to the hotplate.

the lone remaining cable is for the DS18B20 probe and will be shoved down the thermometer well.

use shrink tubing over each bundle and hot air gun blast it. then pass the leads down the breaker box hole on the top and connect them to the respective breakers.
do the same for the enclosure to hotplate, with a Schuko IEC C14 female termination. since the hotplate has to be easily disconnect from the rest, that’s why we terminate we a Schuko. keep this cable and Schuko connector length short.
Screw Schuko connectors are a bit tricky to fasten properly, in my opinion. If I had to do it again I would sacrifice a power cable to use the molded connectors.
repeat the process for the main power AC input, use a male IEC C14 Schuko this time. this cable enters the breaker box through the bottom. keep this cable and Schuko connector length short.

5. Additional resources.

5.1 Firmware

Firmware is available at https://www.github.com/rodv92/milk_heater2

Alpha Version.

5.2 CAD model

Take the model with a grain of salt. it’s mainly for illustration purposes and I used available GrabCAD and Mc Master Carr parts models, some of them do not match the components I’ll used.
the model, as for the date of this post, lacks drilling information, the breaker box is also not modelled.
If one day we have an AI managed database of all market available solid parts, using 3D reconstruction, that will make modelling easier, but human modelers will be rendered obsolete.
AI will drive one of the largest work shifts in human history, that’s for sure.

CAD model uses FreeCAD v 0.2 the model is available in :

https://github.com/rodv92/MILK_HEATER2/tree/main/hw

5.3 Schematics

Also available at (authoritative resources are uploaded on GitHub) :

https://github.com/rodv92/MILK_HEATER2/tree/main/hw/

5.4 Pictures

I hope to keep this project alive so It may serve the makers community well.

3 Phase Modbus Smart Power meter with relay actions and power line communication telemetry

# PMTR_001_SRV_001 V0.22

Description

PMTR (Power Meter, Telemetry & Relay) is an open source project that allows precise monitoring of mains parameters (voltage, current, active power, energy, power factor and frequency) and allows to act on external relays based on formula triggers (ex: overcurrent or overvoltage)It is a client/server project with a Web interface for data display, logging and device configuration.It uses power line communication and the industrial proved Modbus protocol to transfer data between the client and servers; no need to have ethernet or wifi at your main power panel.
It is modular and robust in approach, using off the shelf components available on Aliexpress or other vendor sites to ease replacement of parts if need arise, and thus lower TCO for many years.
The server(s), (PMTR_001_SRV) is an Arduino based device that performs the power measurements and relay actions. Its configuration is pushed by a client.
The client(s) (PMTR_001_CLI) is a Raspberry Pi device that grabs power telemetry and pushes configuration and time synchronisation to the server(s) device. As stated above, it does not rely on an Ethernet connection, just a Power line communication based on the ES-1642 NC module (also available on Aliexpress)
Relay actions are used mainly to protect your equipment (for industrial settings) and control energy costs (ex: hotel industry, power charging for EV), It will also allow to cut/enable power at specific time of the day.
This project, using the modbus protocol, will allow several servers. thus you can monitor individual circuit branchings on power subpanels.
Two flavour of servers are being developped : A single phase meter, and a 3 phase meter. they are rugged using fuses and varistors for overvoltage protection.Metering is based on modified PZEM-004t v3.0 modules that can measure line to line voltages up to 437V AC.
For more information on the client, check: https://www.github.com/rodv92/PMTR_001_CLI

For more code information on the modbus server, check: https://github.com/rodv92/PMTR_001_SRV


Current State of the project.


A single phase proof of concept with a raspberry pi client and a single phase power meter was done and shows good results. The server code integrate formula based actions, and time synchronization. A basic web chart of power parameters and logging into a MariaDB database is part of this POC.


What needs to be done in priority as of November 2022:

  • Finish the 3 phase prototype server, currently designed with Easy-EDA pro.
  • Add ADG333 switching of serial line to query each PZEM-004t V3 modules. (done sequential querying with two modules, ok)
  • Add code for DS3231 time keeping.
  • Harden and test formula actions where conflicting pins are used with different actions.
  • Make the raspberry pi (wifi, serial) configuration seamless
  • Expand the raspberry pi client to query several servers with auto-detection on the PLC bus
  • Add a MQTT layer on the raspberry pi for upstream telemetry reporting
  • keep in touch with PeaceFair, the maker of the PZEM-004t V3.0 to make a hardened 400V module (there are still some issues regarding with high voltage tolerance)

INSTALLATION and SETUP


PMTR_001_SRV IS NOT INTEDED FOR DISTRIBUTION / POWER UTILITY NETWORKS !


typical use case for 3 phase delta-wired measurement:

This use case shows the installation of the PMTR_001_SRV for monitoring voltage and current drawn by a 3 phase induction motor, and all remaining power parameters.


Note that since the motor is a 3 phase delta wired device, PMTR_001_SRV has to be setup in the following way :- Hardware method : By placing measurement jumpers inside the device for line voltage measurement (L1,L2),(L2,L3)(L3,L1)OR- Software method : By specifying a multiplicative correction factor : sqrt(3) for power and energy measurements inside the PMTR web interface. this factor will be uploaded to the PMTR_001_SRV device
The setup contains a single digital pin operating a 5V relay (NO) that provides in turn power to a contactor (NO) that power ups (or down) the induction motor.
It is then trivial to use logic to specify a relay action that evaluates and power ups or down the motor upon specific power conditions. More on relay actions below.
Also note that the PMTR_001_SRV and PMTR_001_CLI are connected on the same line (L1) for PLC communication to occur.

Other installation best practices


PMTR_001_SRV Contains a ES-1642 NC PLC modem (Power Line Communication). No Line filter should be present between the server and the client, the client would be unable to communicate with the server !L1, L2, L3 current coils should be placed on the current carrying wires of the metered installation, not on the wires supplying energy to the meter PMTR_001_SRV can operate over IT, TT, TN-C TN-CS earthing schemes.Being a non metallic chassis, PMTR_001_SRV does not require to be grounded.PMTR_001_SRV contains ceramic fuse protection and varistor overvoltage protection.PMTR_001_SRV will be packaged inside a standar DIN Rail plastic enclosure for ease of installation inside a power distribution main panel.
Check the HW folder of this project for the terminal pin references.

Installation Steps

  • 0) Disconnect power to the facility or to the VFD for safe installation of the current coils.
  • 1) Setup Modbus Address using the 8 position DIP switch to any value between 1 and 254. 0 is a broadcast address. 255 is a bridge address. Do not use these on the PMTR_001_SRV2) Set the 3 jumpers to LN metering – default – (line to neutral) or LL metering (line to line) Usually LN is preferred if the metering is on a standard european consumer network. LL metering is usually reserved for Variable Frequency Drive metering. In that case, skip step 4.
  • 2) After each phase is disconnected from the terminals, insert the current coil over the wire and reattach to the input and output terminals. repeat for each phase.
  • 3) Connect CT- and CT+ wires to the appropriate PMTR-001-SRV terminals. Repeat for each phase Then,
  • 4) Connect neutral wire first to the PMTR_001_SRV
  • 5) You can re-establish power inside the facility now or power the VFD
  • 6) then connect L1 L2 L3. Beware, lines are powered on, use an electrician screwdriver !!!!When connecting L1, the unit should power up, unless on a VFD where you need at least two phases. You can test that all DC power supplies are providing backup power by connecting them individually. PMTR_001_SRV should work with either L1, L2 or L3 connected in LN metering.
  • 7) if a backup / battery supply is available you can connect it to AUX12VDC to the positive terminal and GND to the negative terminal
  • 8) Tie the SEND terminal to any L1 L2 or L3 phase, now the Rasperry Pi (PMTR_001_CLI) can be connected to the same phase and neutral as the PMTR_001_SRV and it should set the time up
  • repeat the same actions 1 to 8 on any other PMTR_001_SRV to install on the network. use the same phase line for all the PMTR_001_SRV that you install
  • PMTR_001_SRV can supply up to 0.8A at 12V over the 12VDC_OUT terminal, electromechanical relays can be powered by 12V IF the 5V relay action pin is recognized as HIGH by the relay.anyway, PMTR_001_SRV can supply up to 1.7A at 5V over the 5VDC_OUT terminal, for electromechanical relays. nb: The relay action pins cannot supply more than 40mA current, use is limited only to SSR relays !!
  • Proceed to PMTR_001_CLI Readme to finish setup.

Modified PZEM-004t v3.0 information


The original PZEM-004t v3.0 can measure up to 267V RMS.
Here, The metering is done through a modified PZEM-004t v3 for up to European phase to phase voltage tolerance :up to 427V RMS with 0.2V resolution. Compared to the non-modified version, This ALSO allows phase to phase metering, and phase to neutral in case of neutral fault.Power, Energy, power factor resolution are doubled due to the fact that their computing involve instantaneous voltage measurement which is affected by the PZEM modification.
Otherwise, the floors, ceilings of the remaining parameters conforms to the PZEM-004T v3.0 datasheet of this day.It is expected that frequency measurement is not affected by the PZEM-004t v3.0 HV modification.For power factor, leading or lagging info is not provided.
PMTR_001_SRV performs instantaneous measurement as fast as the 9600 bps serial interface allows, which is a little less than 100 per second (all parameters read)When performing 3 phase metering the PMTR_001_SRV performs serial port switching (using an ADG333A Quad SPDT switch) So it shares a single serial port on the Arduino for 3 PZEM-004T modules.this lowers the read rate to less than 33 per second.
an average value over 1 second of these 33 samples is stored as the “1 second average”On the basis of this 1 second average, then a moving average from 2 to maximum of 60 seconds is saved.
The device has 5 memory slots for Moving Average information of all parameters (except energy)For instance, We could store the Moving Average over 5, 10, 20, 30, 60 seconds.
The incentive behind moving average calculation is to apply low pass filtering and filter any spurious spike that would make an inappropriate relay action at the time.

Relay Actions


A ‘relay action’ is a set of the following parameters :

  • An expression : using tokens representing PMTR_001_SRV parameters, and a set of inequalities and logic.- A moving average to use for expression evaluation (except energy which is never averaged)
  • A dry run bit (Yes/No), which tells whether to perform or not the relay action, while still logging all action behaviour
  • A recovery period during which the expression must keep evaluating to false before reverting relay action. This hysteresis is implemented for protecting devices which should not be turned on/off too frequently.
  • A HIGH or LOW state to command the relay when expression evaluates to true. (on false evaluation the state commanded will the be opposite ) nb : This evaluation from memory is the first action to be executed at boot up.
  • A list of pins on which DIGITALWRITE Will be performed.It is recommended that each action activates a separate set of pins, otherwise the last evaluated actions will have precedence and overwrite the status of pins.


There are 5 actions slots.

Examples of valid expressions :

  • (L1U > 253) || (L2U > 253)) || (L3U > 253)
  • LII > 16
  • LIp > 1200
  • L2e > 10
  • (L1f < 49.1) && (L2f < 49.1) && (L3f < 49.1)

Ex : L1U is phase to neutral L1 RMS voltage, L1I is phase to neutral RMS current, L2f is L2 frequency.Units use are SI, excepts for energy which is in kWh
First case would be the detection of a voltage swell over 253V over L1,L2 and L3 at the same time.The second a current of more than 16A on L1Next an active power of more than 1200W on L1Next, A total energy consumed of more than 10kW over L2. Finally a frequency sag of 49.1 Hz over any of the phases.

Neutral fault detection

Broken neutral (floating) fault in three phase systems can have disastrous consequences by essentially rewiring the loads in a delta fashion.

Depending on the location of the fault and earthing arrangements, voltage imbalances arise, and the neutral point is shifted and may be subjected to voltages relative to earth that may be unsafe.

A sucessful neutral fault detection without false positives is not trivial based on a limited number of voltage readings (three power meters only), absence of phase angle information, and absence of a neutral to separate ground electrode voltage measurement

However, using magnitudes of voltages only and their variation in the time domain according to load variation, we can tentatively propose an lightweight algorithm suitable for MCUs to detect such occurences.

Keeping in mind that if the load are reasonably balanced the fault will be harder to detect, until a disturbance in the balance of loads will unbalance voltages.

Generally speaking, a broken neutral condition will translate in an increase of one phase voltage, while the other two will decrease with reference to the nominal value, or one will decrease while the other two will increase.

A lightweight pre-check for discriminating between single phase power disturbances and neutral fault would be to check for any voltage exceeding a 10% nominal value deviation, and then proceed to check remaining voltages for disturbances.

A valuable tool for analysis of neutral fault conditions is the three phase voltage triangle. By using basic trigonometric identities of the equilateral triangle, we can characterize the least possible deviation in absolute values of the other two voltages assuming epsilon is the deviation factor from nominal value for the upmost voltage.

$$ \epsilon = 1.1 $$

$$ V_{p} = 230V $$ is the nominal phase RMS voltage

$${V_p}^{‘} = \frac{V_p \sqrt{4+12\epsilon^2}}{4\epsilon} $$ is the least possible deviation from nominal values for the two remaining voltages.

Thus we can write : $$ V_pdev >= |V_p – {V_p}^{‘}| $$

So, in terms of ratio we have $$ \frac{V_p^{‘}}{V_p} $$

The algorithm has to check that the remaining two voltages is higher or equal to that ratio for the first voltage and lower or equal to that ratio for the second voltage. Once this pre-characterization is done, a second part, more computationally intensive part of the algorithm can kick-in, to check that the sum angle derived from magnitude of the phase voltages is still close to 360° degrees, while analyzing voltages changes in the time domain.

Phase angles can be derived from magnitudes using sine and cosine trigonometric identities of the triangle. voltages fluctuations on the three phases with time that keep their sum angle close to 360° can be assigned a score for each angle sum characterization. With a value close to 360° giving a positive score, and a value diverging from 360° giving a negative score.

A sum of absolute values differences between discrete time samples of the three voltages can assign a weight to the characterization. This characterization puts into evidence the variation of load impedance in a broken neutral condition, and gives more credence to the computation if the angle sum keeps getting close to 360°

Once a positive score compounded by the previous weight reach a threshold, a broken neutral condition can be raised.

There are caveats however. a robust algorithm must assess the stability of the line voltages over time, preferably over a long term period. Using only three power line meters, a common occurence of unstable line voltages will degrade significantly the predictive abilities of the algorithm.

Hardware in-depth information

AC protection features


– 20mm x 5mm 0.1 A ceramic fuses, fast blow 250V, this for each phase L1 L2 L3 and neutral N- 14D481K varistors between each phase and neutral, and for protection of the PLC module.

AC/DC conversion

Power is provided by YHT4S12V/10W modules AC380V tolerant.the three modules are connnected line to neutral and should be tolerant to neutral fault and up to 2 phase faults.


DC Bus

Matched Schottky diodes are used to improve power sharing betweeen the modules at 11.5Va single LM2596 dc step down module provides 5V to the DC bus

PLC

ES-1642NC uses a separate hardware serial port for modbus communication over one phase and neutral. the serial line speed is 9600 bps. the PLC speed is between 2.5 to 4.5 kbps. Check Taidacent product page for more information.

MCU


The main controller is an Arduino mega 2560 it is on a separate mezzanine board than the HV parts / PZEM Modules.
The following IC : ADG333 allows routing a single hardware serial port to the three PZEM modules. Querying is continuous and sequential (all parameters of module 1,2,3,1,2…)
It could be technically feasible to have all PZEM modules on the same RX/TX serial TTL pair, but that would need to ensure that the optocouplers do not overload the RX/TX current sourcing capabilities, and also would need a factory preset of the PZEM-004t V3.0 addresses. Note that PZEM-004t V3.0 itself is a modbus device.
The server itself encapsulate telemetry queries and present the 3 PZEM-004t V3.0 as a single modbus server device to the Raspberry Pi client.
Thus, it seems more adequate to use the same Modbus address on each module, and switch them using an IC switch like the ADG333.

Time keeping & Synchronisation

A real time clock (RTC) 3231 mini module is used to keep track of time and the synchronization is done over PLC / Modbus.
Modbus address encoding of the server is done throught a 8 bit DIP switch encoder.

Software (check github repository)

Most of the single phase functionality is POC tested.three phase hardware testing and harderning is still in progress.Software is still in POC phase. Consider the code to be in alpha stage.

https://github.com/rodv92/PMTR_001_SRV

https://github.com/rodv92/PMTR_001_CLI

Hardware Schematics

Logic Board – Low Voltage
This is the logic mezzanine board.

Power board – High Voltage
HV PCB Layout


Software library includes & Credits

Modbus library used for PLC client/servers communication ArduinoModbus depends on the ArduinoRS485 library https://github.com/arduino-libraries/ArduinoRS485https://github.com/arduino-libraries/ArduinoModbusby sandeepmistry

specific modbus conditioning and parsing library for the power monitoring moduleshttps://github.com/mandulaj/PZEM-004T-v30by mandulaj
library for timekeeping https://github.com/PaulStoffregen/Time/blob/master/TimeLib.hby PaulStroffregen

library for scheduling task by timer ISRshttps://github.com/khoih-prog/TimerInterruptby khoih-prog
library for parsing math and logical expressions (for relay logic)https://github.com/zserge/expr/blob/master/expr.hby zserge

The Synth1 project

For the MCU code, please visit : http://www.github.com/rodv92/synth1

For the Circuit / PCB design (work in progress) please visit :http://www.easyeda.com/rodv92/synth1

A Digitally Controlled XR-2206 based VCO

 

Abstract

 

The goal of this project is to implement a XR-2206 based VCO with digital control of frequency, amplitude, wave shape, and duty cycle, maximizing the use of on-chip capabilities of the XR-2206.

This project may be used as a standalone function generator. A MCU is used for tuning of the aforementioned parameters. Hence, This is why it is said to be « digitally controlled » compared to manual potentiometer control.

The incentive for using an analog XR-2206 VCO instead of a simpler DDS (Direct Digital Synthesis) IC is to offer a solution that exploits the unique feature of the XR-2206, as waveform symmetry adjustment and THD control.

In some situations, injecting an analog signal with its artifacts may be of interest.

As an analog oscillator, the XR-2206 is free from digital aliasing artifacts.

It is also a good choice for musical instrumentation as an analog synthesizer, as it’s unique features (symmetry adjust / THD control / wave-shaping from sine-wave to triangle wave) give warmth to the generated sound, which is sought after in the analog synth market.

It also allow a faster development than a full-on discrete analog oscillator design.

 

 

Presentation of the XR-2206 IC

 

The XR-2206 is a VCO containing two sub-oscillators, with a keying pin allowing the selection of either sub-oscillator for output on two signal output pins, one presenting a sine/triangle/saw shape, the other pin being the output of a square wave signal.

Frequency Control

Current controlled

Coarse frequency control is achieved through the use of a given resistor to ground from the frequency current control pin. Each oscillator has its dedicated frequency control pin.

Frequency range (common to both oscillators) is further achieved through the use of a capacitor on the capacitor timing pins.

The resistors and capacitor are further referred as « timing resistors » and « timing capacitor ».

The resistor pins are further referred as timing current pins.

This basic frequency control achieved in this manner is dependent on the current flowing through the resistor timing pins, hence, this setup gives, strictly speaking, ​​ a current controlled oscillator for frequency selection.

The data sheet states the minimum value for the resistors being 1K, in order not to exceed the safe margin for current sourced from the timing pins.

The formula for the frequency of oscillation for each oscillator being 1/(R_subosc*C), C being the capacitance of the frequency range capacitor common to both oscillators.

The timing current pins are internally biased at +3V

A coarse MCU control of frequency can be achieved in this manner through the use of digital potentiometer wired as a rheostat.

Renesas X9C10x series were demonstrated to work well in this setup, as the voltage to ground is inside the specification range, When using X9C102, a trimming rheostat must be included so the current sourced from the pin does not exceed the safe specification margins of both the digital potentiometer and of the XR-2206. (values??)

A more granular current control scheme can be achieved by using the digital potentiometers X9C104, X9C103, and X9C102 as rheostat in series to ground to achieve at 10 ohm step.

However, this approach has drawbacks :

X9C10x series have up to 20 % tolerance in total R_High to R_Low values, although the resistance value of each R step on the R ladder demonstrates good linearity.

Thus, It is imperative to carefully select matched digital potentiometers for both sub-oscillators and measure precisely the resistance of each digital potentiometer so that the MCU can tune the frequency with minimum readjustments.

Also, the use of multiple digital potentiometers in series makes the frequency tuning control algorithms more complex and each new potentiometer introduces error coming from the resistance measure and makes the time to tune longer. Careful design must be implemented so that out of tuning bounds events for each potentiometer do not happen, which would further delay the time to tune.

For all these reasons, a single digital potentiometer in rheostat mode for coarse tuning plus a DAC chip in voltage control mode for fine tuning is preferred as it reduces the number of digital control channels, while maintaining a large frequency range and good frequency resolution, while minimizing tuning time, but increasing BOM costs.

Nevertheless, if tuning time is not a priority, a lower cost hardware setup with higher control complexity algorithm can be devised.

Example of operation for an audio range frequency current control setup with three X9C10x digital potentiometer in series at each R timing pin.

In this setup, a polypropylene capacitor of 0.044 µF is inserted on the frequency range capacitor pins.

Provided that a trimming potentiometer ​​ in rheostat mode, is inserted in the series with one X9C104, one X9C103, and one X9C102 digital potentiometer in series, the minimum R_trim value setting the highest achievable frequency will be set so that the lumped resistance is no less than 1KΩ, stated by the XR-2206 datasheet as the minimum resistance for safe operation.

Additionally, the XR-2206 datasheet shows exponential degradation of frequency stability at large (>200 kΩ) and low (~1kΩ) timing resistors values. Thus, it is not advisable to use digital potentiometers with a total lumped resistance above 200 kΩ. This is the primary factor restricting the range of output frequencies.

Obtaining several output range frequencies can be done by selecting different capacitors of a bank in 1 of n fashion.

The X9C10x series was selected for its low cost, up to 7V voltage maximum rating and current rating in the low mA range. It also features recall of last setting at power-up, if needed, with a durability of 100000 cycles per bit of the recall logic.

However, it does not offer logic query of the current setting. That means that the MCU should maintain the potentiometer states in RAM and EEPROM if a recall of the setting at power-up is required. However, in-board MCU also have a cycle write limit for EEPROM, thus a flash memory storage would be preferable for frequent updates.

In the case of a musical instrument, the settings of these potentiometers affect frequency only, not timbre, and thus can be discarded at power-off.

Determination of frequency range

On the X9C10x datasheet, it is stated that the minimum resistance of the digital potentiometer is 40Ω, corresponding to the wiper resistance, noted RwiperR_wiperin the following formula defining the minimum value of the trimming resistance based on three digital potentiometers in rheostat mode used in series :

3Rwiper+Rtrim>1kΩ3*R_wiper + R_trim > 1k %OMEGA

solving for RtrimR_trim, supposing RwiperR_wiper​​ being close to the nominal 40 Ohms value.

Rtrim>880ΩR_trim > 880 %OMEGA

And, with C set at 6.6e-8 F (0.066 µF),

We can compute the maximum frequency we can obtain :

fmax=110006.6E8f_max = {1} over {1000 * 6.6E^-8}

fmax=15151Hzf_max = 15151 Hz

The lowest achievable frequency, provided that the potentiometers are each at nominal values of, respectively, 100 kΩ, 10 kΩ and 1 kΩ, and set at RmaxR_max, so that Rmaxtotal=100k+10k+1k+880+340R_maxtotal = 100k + 10k + 1k +880 + 3*40  :

Rmaxtotal=112kΩR_maxtotal = 112k %OMEGA

fmin=1112E36.6E8f_min = {1} over {112E^3 * 6.6E^-8}

fmin=135.8Hzf_min = 135.8 Hz

 

In the following implementation of a VCO for a musical application, as a synthesizer, RtrimR_trimwas offset at a higher value so that :

fmin=f(C3)=130.8Hzf_min = f( C3 ) = 130.8 HzThe three digital potentiometers being set at full-scale (99,99,99)

Based on the aforementioned relation, We can solve for RtrimR_trimand RmaxtotalR_maxtotal

So that,

Rmax=115,84kΩR_max = 115,84 k %OMEGA

Rtrim=4.717kΩR_trim = 4.717 k %OMEGA

 

Note that by doing this, fmaxf_max​​ is then offset to a lower value of 3132 Hz.

Which gives us a frequency range close to 4.5 octaves. (C3 to F#7)

Frequency tuning step resolution

 

Since the three digital potentiometers have 100 equal R steps (from 0/99*r_max to 99/99*r_max) and are fully overlapped, in the sense that no R setting is not accessible int the ​​ [Rmintotal,Rmaxtotal][R_mintotal , R_maxtotal], the RstepR_step​​ of the lowest resistance digital potentiometer (X9C102) gives the resistance step resolution, that is,

Rstep=1E399R_step = {1E^3} over {99}

Rstep=10.1ΩR_step = 10.1 %OMEGA

Which gives a frequency resolution at fminf_min​​ of :

Δfmin=1(115.84E310.1)6.6E81(115.84E3)6.6E8%DELTA f_min = {1} over {(115.84E^3 - 10.1)*6.6E^-8} - {1} over {(115.84E^3)*6.6E^-8}

Δfmin=0.0114Hz%DELTA f_min = 0.0114 Hz

and at fmaxf_max:

Δfmax=1(4.837E310.1)6.6E81(4.837E3)6.6E8%DELTA f_max = {1} over {(4.837E^3 - 10.1)*6.6E^-8} - {1} over {(4.837E^3)*6.6E^-8}

Δfmin=6.5544Hz%DELTA f_min = 6.5544 Hz

In terms of musical units, the interval compared to C3 at fminf_min​​ is :

Δfmincents=1200log2(f(C3)+Δfminf(C3))%DELTA f_mincents = 1200*log2({f(C3) + %DELTA f_min} over {f(C3)} )

Δfmincents=0.1508cents%DELTA f_mincents = 0.1508 cents

And at f_max = 3132 Hz

Δfmaxcents=1200log2(3132+Δfmax3132)%DELTA f_maxcents = 1200*log2({3132 + %DELTA f_max} over {3132})

Δfmaxcents=3.619cents%DELTA f_maxcents = 3.619 cents

Dynamic Tuning Algorithm

Single Sub-oscillator tuning

A fully digital algorithm based on the Arduino Mega 2560 was devised.

Measuring the frequency was achieved through the improved resolution timer2 library and a modified version of the PulseIn that integrates the 0.5µs precise timer2 library instead of the 4µs standard precision.

It should be noted that accurate frequency measurements are dependent on the clock frequency of the Arduino. Arduino mega 2560 is clocked at 16 Mhz, However it uses a ceramic oscillator instead of Quartz crystal oscillator. Thus, it is probable that the oscillation is not an exact 16 Mhz.

This is not a problem for interval tuning since it is based on frequency ratios, but may introduce de-tuning if the synthesizer signal is to be integrated into an ensemble of other instruments using, for instance, a reference tuning of A4 = 440 Hz.

Also, The higher the working clock frequency of the MCU, the higher is the measure resolution. and tuning speed. However, the limiting factor in the tuning speed achieved is the period of the signal. Measuring the frequency is usually done through signal level or edge triggering or zero cross  ​​ ​​​​ method, all requiring at least a lapse of time of 1/f_fundamental to compute the frequency of the signal.

In practice, oscillator drift, noise, may require averaging several frequency measurements to obtain a reliable measure.

In this implementation, the number of frequency measure sample has been set to 10.

An analysis of the statistical distribution of frequency measures over a medium interval (like 10s) will give an idea of the effects of noise, digital aliasing artifacts. Oscillator drift is usually temperature dependent and should be minimized by using an adequate capacitor with a flat temperature to capacitance response. (temperature drift compensating)

In this implementation, low ESR metalized polypropylene capacitors are used, for their adequate frequency response, long lifetime and parameter stability over time.

 

The first step in tuning uses the f = 1/RC formula, solved for R. It is important to have a precise measure of the timing capacitor capacitance at standard temperature conditions of use.

R = 1/fC.

Now, three values of resistance should be determined satisfying the following conditions :

R1 + R2 + R3 = R – Rtrim - Rwipers

R1 < 100K

R2 < 10K

R3 < 1K

Ideally, R2 and R3 values should be as close as possible to the mid-range of the digital potentiometer, that is, 10K/2 and 1K/2. This gives leeway for fine tuning of R2 and R3 in both increasing and decreasing R steps.

Based on the above formula, the three digital potentiometers are set to their respective values, and the sub-oscillator frequency is measured.

The frequency error is then : ferror=ftunedfmeasuredf_error = f_tuned – f_measured

The goal being to tune the oscillator at the R value of min(|ferror|)min(abs{f_error})

A standard gradient descent approach based on the derivative of frequency over resistance function is used dRdf=1f2C{dR} over {df} = {-1} over {f^2 *C} so that the step delta_R for each R correction is given by :

 

 

ΔR1f2Cferror%DELTA R simeq {-1} over {f^2 *C} * f_error

In practice, a tuning factor k is introduced and determined empirically in case of undershoots /overshoots, k < 1 in case of overshoots and k > 1 in case of undershoots

ΔR1f2Cferrork%DELTA R simeq {-1} over {f^2 *C} * f_error * k

Considering a simple algorithm that would make corrections on the digital potentiometer R values sequentially, starting with the highest value potentiometer for coarse tuning. The potentiometer step change number is given by the rounding to the nearest integer function [][<?>] ​​ :