TCL = Tool cmd language

Tcl is similar to other shell lang. TCL can be scripting language, but it can also be used as shell as "tclsh".

TK can be used to develop gui i/f.
Tcl/Tk is open source, and can be included in C/C++ etc, or vice versa.

official tcl/tk documentation (very detailed): http://www.tcl.tk/doc/

As alwyas, tutorial point is very good source for learning anything: https://www.tutorialspoint.com/tcl-tk/tcl_tk_quick_guide.htm

TCL is a very weird language, interpretor written ver differently than how other high level lnguages are written in. Richard M Stallman has openly said that Tcl as a language should be completely avoided. There are weird cases, and language is more of learning by experience than learning by syntax. Same code written in 2 ways doing the same thing and syntactically correct, may be interpreted by the interpreter different and give different results. You should avoid using Tcl for any general purpose work, and use it only where it's absolutely needed for the job.

TCL is mostly used in CAD tools. If you are going to be using any hardware CAD programs from Synopsys, Cadence etc, they exclusively use tcl. So, if you have to learn it, whether you like it or hate it.

Tcl/Tk Installation:

tcl/tk may not be installed by default on most Linux OS. Here's the steps for downloading and nstalling tcl/tk:

On CentOS: With any software installation, there are 2 ways to install. Install it using package installer (which will automatically download and install all dependencies), or download it yourself and install it. The preferred way is using "yum" package installer on CentOS, but sometimes that package is not available via yum, so we may be forced to do manual process.

A. yum installation: sudo yum install tcl => This shows tcl.x86_64 as the pkg being installed. Click "Y" and it installs it in /usr/bin/tclsh (using cmd "which tclsh" shows it). For some reason, this wasn't working well with some other tool (missing files), so I installed it directly

B. direct installation: Download it from "https://www.tcl.tk/software/tcltk/download.html". I downloaded tcl8.7a1-src.tar.gz and tk8.7a1-src.tar.gz. After downloading it in some dir, goto that dir using gui file browser, right click and choose "extract". This will unzip and untar all files/dir and create new dir with same name "tcl8.7a1" and "tk8.7a1". Once this is done, goto unix terminal. 

1. cd tcl8.7a1/unix.

2. type "./configure"

3. type "make" => this will finish, with no "success" or "fail" info. Just some "gcc -O2 ..." cmds at the end.

4. type "make test" => runs all tests. Summary should show all passed (Total=31405, Passed=30101, skipped=1303, Failed=1= http.test). Many will be skipped. That's OK.

5. type "sudo make install" => installs it in "/usr/local/bin/tclsh8.7" All files are in /usr/local. Lib are in /usr/local/lib/tcl8.7/

6. If we run tclsh, it won't find anything. Even though "echo $PATH" shows /usr/local/bin in the path variable. Reason, tclsh is installed as tclsh8.7. We will fix it in step 8.

7. Now repeat same process for tk8.7a1. cd unix, ./configure, make, make test (Total=9544, Passed=8459, Skipped=996, Failed=89,Then again shows Total=465, Passed=448, skipped=17, failed=0), sudo make install. This gets installed as /usr/local/bin/wish8.7

 8. Now create soft links so that tclsh (which is actually tcl) and wish (which is actually tk) can run by typing tclsh or wish. cd /usr/local/bin. "ln -s tclsh8.7 tclsh" and "ln -s wish8.7 wish". Now all the tools which rely on tclsh or wish should be able to access it, as long as the path "/usr/local/bin" exists in PATH var. tclsh and wish are installed with their version numbers, so that different versions of the same lang can be installed at the same time without overwriting each other. That way if 2 different tools require 2 different versions of a pgm, they can run simultaneously by changing soft links or hardcoding the actual path of pgm with correct version number.

NOTE: when havving multiple versions of tcl/tk on your system, some files do not get installed correctly, resulting in "missing files" when calling an app that requires tcl/tk. Part of the confusion may be because some files are named same for for any version of tcl/tk installation resulting in 1 file getting used for mutiple versions, and depending on which is the last one that got over written, it may or may not work with other versions. Make sure you have these files:

> whereis tk
tk: /usr/lib64/tk8.5  /usr/include/tk.h /usr/share/tk8.5
> whereis tcl
tcl: /usr/lib64/tcl8.5 /usr/include/tcl.h /usr/share/tcl8.5

> whereis tclsh
tclsh: /usr/bin/tclsh /usr/bin/tclsh8.5 => tclsh is just a link to tclsh8.5

Depending on manual installation, these files might also be in /usr/local/bin/ and /usr/local/lib/ (since all software gets installed in /usr/local/ by default). Make sure, these files are there:

/usr/share/tcl8.5/ => has tclIndex, bunch of other tcl files and dir

/usr/lib64/  => has tclConfig.sh, libtcl8.5.so and tcl8.5 dir (this dir has tclConfig.sh which is just a soft link to tclConfig.sh in /usr/lib64/).

/usr/include/ => has tcl.h and bunch of tcl*.h files and dir


TCL syntax:

Any language has data and cmds (or operators) to act on data. Then there are special characters. All these are parsed by the interpreter to figure out what to do with that line to produce the result. We'll learn about data (var and their data types), cmd (reserved keywords and operators) and special characters (anything other than alphanumeric and operators).

tcl is case sensitive lang.

tcl script is a string consisting of 1 or more cmds. Tcl cmds are list of words, with first word being the cmd, and remaining words being args. This is totally different than how all other scripting languages or high level languages work. Whitespace separates multiple words of the cmd. To group multiple words into single arg, we use {} or "". To separate different cmds from each other, newline is used (so use separate line for each cmd). To put multiple cmds in same line, use semicolon ( ; ). So, newline and semicolon are very important in tcl (you need atleast 1 of them), else diff cmds may not be recognized.

NOTE: Since each cmd in tcl is on it's own line (or separated by ;), you can't put characters as {, }, ", etc anywhere. You have to follow strict syntax so that they get recognized as a group. So, tcl is very different than other languages where spaces, newline and positioning of characters matter. This is due to the fact that every string is treated as a cmd. So, if the string at the start of the line by itself doesn't seem to be a cmd, tcl will error out with "unkown cmd" (i.e having { by itself on a line is not a valid cmd). See more examples below with various cmds.

This link neatly explains how tcl interpreter interprets words: http://www.tcl.tk/man/tcl8.5/TclCmd/Tcl.htm

 

Tcl special characters: Just like all other languages, tcl has it's own set of special characters. They are mostly consistent with shell (bash, csh) special characters, but differ in some ways.

  • $ : dereferences a variable. $a refers to contents of var "a"
  • ( ) : Used for grouping expressions. This is used with arrays. It's not used in grouping cmds or words, as in "for" loop, etc. Curly braces { ... } explained below are used for all those cases. So, use of () is very limited in tcl.
  • [ ] : This does cmd substitution. Denotes a nested command. these evaluate cmds/scripts inside [ ]. The result is then put out and used by other cmds surrounding this. It means evaluate that string as a script immediately and use the result of the script as the value to substitute. It's used when calling user defined proc or other built in cmds.
  • \ : Used for escape quoting. Any char after \ is treated as ordinary char (loses it's special char meaning), and included in the word. Escape rule is similar to \ in bash, where it hides all special char from shell, but only 1 char at a time. The exception to this is the following special sequence => \a, \b, \t, \n, \\ ( this gets substituted as backslash \ ) , \xhh, \uhhhh,. So, \n is not escaped as tcl compiler sees the word "n" after "\" and omits escaping. However \\n will be seen as escape "\" followed by "\" and then a letter n, so final escaped value would be "\n". Escaping still happens within double quotes "..." but not in braces {...}. See next 2 bullets. 
    • If you put \ at end of line then since backslash escapes whatever comes after it, it will escape the newline. This looks similar to \\n, but here "\n" is one char (newline char), so that whole newline char is escaped. So, the shell doesn't see it as end of line. That means that next line will be treated as continuation of current line, and keeps on doing it for as many lines as you have \. All of these lines are then treated as one string.
    • VERY IMP: If you put space after a "\" at the end of current line, then \ escapes the space and not the newline. Escaping the space does nothing (since space is not a special char). Since newline is not escaped, the current line ends and new line starts from next line. This may break your code, as you may have intended for next line to be continuation of current line. Next line may not be valid cmd anymore and you may get "error". This is very common mistake in tcl, and since most editors don't show space explicitly, be extremely careful when using "\" at end of line. This is an issue in comment line starting with # too, see below for details.
  • " " : Denotes weak quoting; Command substitution, variable substitution, and backslash substitution are performed on the characters between the quotes. Groups multiple words into one. This is similar to  double quotes " " in bash, where only whitespace char are hidden from tclsh. It treats groups of words inside " " as 1 string.
    • ex: puts "..\n... \$a..." => Here newline char is not escaped (as it's excluded from escaping anyway(see escape quoting above). $ sign is escaped here even though it's inside quotes, as backslash escaping is still done inside quotes. So, we print $a instead of value of $a as $ got ecaped.
  • { } : Denotes rigid quoting; there are no substitutions. Groups multiple words into one. it means “don't do anything with this string inside { } at all; use it as-is. Sometimes you're feeding that string into a command that evaluates, so this helps do that, as nothing gets substituted. This is similar to single quotes ' ' in bash, where all special char are hidden from shell. The second most popular use of { } is to group words everywhere, i.e in loop body, conditional stmt, proc, etc. However, strictly speaking, they are not required at all except for places where we don't want substitution. We use them in proc args/body, but we can write proc entirely w/o using curly braces. curly braces within a comment are considered too when looking for start-end pair of {...}, so be very careful to not have single curly braces inside a comment (see below in comment section).
  • ; : Ends a command. needed when multiple cmds are used on same line
  • # :Begins a comment. Comments may be placed at start of line, or after end of cmd (which ends by having a ; at the end. Comments in Tcl are just not ignored, but are put in a "process later" bucket while the parser keeps moving forward. In tcl, you just can't take a piece of code, and comment it using "#" expecting it to run without errors. Syntax errors within "#" may still give errors. 
    • ex: set a 2; #my comment here => This works as a comment
    • if {0} { all comments } => #comment has many caveats as shown below, so one popular alternative to using # is to put comments in this "if 0 { ... }" construct (braces not necessary around 0). This always works without issues.
    • CAVEAT 1: braces {...} should always appear in pair in a comment, else it will give rise to mismatched { ... }. {...} in comments which are not in a pair will be errored out by the interpretor as not having matching braces. Also, the interpreter will point you to "error" in start of the program or in other weird places, when the error is due to mismatched { ... } in comments. It has no clue of where the mismatch happened, when it's nside a comment.
    • CAVEAT 2: comments in tcl are not just ignored, but are also looked for "\" at end of line. This is in contrast to bash/csh, where they are just ignored. So, if you comment out an existing piece of code which has \ at the end of line, your comment may not work. This is a pain, as comments can't be put blindly over a piece of code. One side benfit of this is that a comment can be extended to other line by using "\" at end of 1st comment line. This gives rise to some weird code as seen below:

ex: sometimes we see code where we execute a script2 within a shell script1, which sources the same original script1. This looks like a recursive infinite loop, but can work well using "\" at end of comment for tcl files. This script myscript.sh below is a bash shell script. Only 1st 3 lines are bash cmds, remaining are tcl cmds. On running, this script starts running as bash, based on 1st line, ignores 2nd line which is a comment (ignores \ at end of comment), and runs 3rd line "exec" cmd. That calls a pgm pt_shell which has a source $0 in its arg. pt_shell replaces the current shell and sources myscript.sh (which is the original file itself). However, pt_shell is a pgm that only sources tcl file, and starts parsing it per tcl syntax. It sees any line starting with # as comment . So, it ignores first line as comment (even though tclsh would have interpreted that, but looks pt_shell just ignores it). It looks at 2nd line, sees continuation of it on 3rd line (due to \ at end of 2nd line), and so considers 2nd line as comment (with 3rd line included in it). Then it moves to 4th line and start reading the file normally. So, here we are able to use the same file as both script and source file.

#!/bin/sh

# \

exec pt_shell "source $0"

other tcl cmds ....

variables:

Just like any other language, we define variables in tcl. Variables can be any data type. Tcl has it's own data type explained below. variable names need to conform to a syntax. In tcl, these var names or identifiers start with letter or _, followed by letters, digits, $,_. ex: a123_. However going thru tcl doc, it looks like a standard scalar variable can be comprised of any character. However the variable substitution operator "$" assumes you are using alphanumeric characters and underscores (plus name space separators "::"). So, we stick with _ and alphanumeric char for var names, so that they work w/o issues when used with $. To assign a value to a var, "set" cmd is used. To print value of any var, "puts" cmd is used. syntax of various cmds is explained in cmds section below.

ex: set a123! me => even though a123! has special char "!" in it's var name, tcl accepts it. It set var "a123!" to value "me"

puts $a123! => errors out, since $ assumes that var name is only _ and alphanumeric char. So, the first time it sees "!", it assumes that var name is complete and so var name is a123. ! is assumed to be start of next word. So, it tries to print value of var "a123", which obviously doesn't exist. Gives ERROR: can't read "a123": no such variable

If we put var inside curly braces, then no substitution is done, and $ works correctly.

puts ${a123!} => prints "me", since a123! is not substituted, but treated literally. So, $ sees a123! as var name. This technique also used when printing var concatenated to other char as "/a/b/${myvar}_c/d". Here, without curly braces, $ will look for var name "myvar_c" which doesn't exist.

unset a123! => This unsets the var so that it isn't defined anymore. However, if var a123! doesn't exist, then it will error out. So, the var needs to be set before it can be unset.

NOTE:  TCL is used almost exclusively with all CAD software to write scripts to be run on the CAD software. One thing to keep in mind is that when you run the same script multiple times or different scripts in the same session, var definitions from one iteration of the script or from multiple scripts are saved in memory, so the value of var from prior script or different script may still be used in current run. In such cases, the script may fail when run on a different session. Or you may get errors like "var myvar is not an array", when you want to set an array. This happens because var may be defined as normal var in previous iteration, and in latest run you are trying to make it an array, which conflicts with previous defn. In such cases you should always unset all var at beginnng of script, so that script starts clean. One issue is that var needs to be set before it can be unset. To get over that use catch.

ex: catch {unset a123!} => Now even if the var is not defined or set, it will not error out.

------------

In TCL there are many reserved words that can't be used as identifiers. ex: for, if, expr, etc. Also, tcl special var explained below can't be used as identifiers.

tcl special var: have predefined usage (i.e tcl_version, env, argc etc)

  • puts $tcl_version => gives 8.6
  • puts $env(PATH) => prints PATH = /home/cg/root:/..../:
  • argc = num of cmd line args, argv = list of cmd line args, argv0 = name of script being invoked

whitespace: whitespace (blanks, tab, etc) is needed in tcl to separate parts of stmt so that interpreter can identify them. Sometimes, it's not necessary, i.e [expr 3+2] will work correctly even though 3+2 has no whitespace. Recommended to use whitespace everywhere, so that there's no confusion.

Example script in tcl:

run standalone:

ex: hello.tcl
#!/usr/bin/tclsh => tells this script should be interpreted using tclsh.
# comment is here => comment. multiline comment would be using "if 0 { comments }". We do not have C style of multi line comment.
puts "Hello World!" => io cmd. puts is cmd, "Hello World" is arg. no semicolon at end. semicolon is needed if inline comment is used => puts "abc"; #comment.
puts stdout "Hello" => here 2 args provided. prints Hello on screen (std o/p)

run: ./hello.tcl or tclsh hello.tcl. If 1st line #! is not there in tcl script, then ./hello.tcl doesn't work (as current shell doesn't know what interpretor to use to run this file). Note: extension of file doesn't have any meaning in linux.

run interactively:

Instead of writing script, we can run tclsh in interactive mode by typing tclsh. Then shell changes from whatever shell you are currently in to tclsh. You can type tcl cmds in that shell, and it will show corresponding outputs. To exit tclsh, type exit and enter.

ex: [/home/ajay]$ tclsh => causes it to switch to tclsh
% expr 2 + 3 => n tclsh, we issue tcl cmds
5 => result of tcl cmd

% exit => cmd to exit tclsh

[/home/ajay]$ => after typing exit above, we are back to bash or csh


Data and Cmd:

Data may be integer, string,etc and have a identifier name (as my_name, etc). Cmds (or operators) are +,-,while, for etc. There are reserved words that are used for cmds which can't be used as identifier names. In many languages (eg C) data types need to be defined as to whether it's an integer, float, string, etc, while in most interpreting/shell languages (eg tcl) data types do not need to be defined, and are figured out automatically by the interpretor. This can be a source of confusion, as we do not know if a particular variable is treated as string or number in a particular operation. Many times, it's clear as some operators only operate on particular data tpes (i.e + only works on int and float), but sometimes it's more complex.

Data types: primitive data type is string (tcl is called string only lang). Everything is a string, and converted to other types as needed. No declaration needed. There are 2 composite (or derived) data types: list and array. So, this keeps the languauge very simple. Tcl can auto transform one data type to another when needed. However this makes the lnguage slow, as adding 1000's of numbers will require converting numbers from strings to number (as all numbers are by default treated as string, and then interpreted to be numbers which requires expensive conversion).

Following are the data types in tcl:


1. string: This is the only primitive data type. string can be put w/o quotes (for single words), with double quotes (for multiple words where we want weak quoting, i.e cmd, var and backslash substitutions allowed), or with curly braces (for multiple words where we want strong quoting, i.e no substitutions allowed).
ex: set myVariable 18 => stored as string, even though no quotes used, since it's single word.
puts [expr $myVariable + 6 + 9] => myVariable is converted to integer and 33 is returned, which is again stored as string and passed on to puts, for puts to print it.

ex: set me "John Hass" => double quotes used since it's more than 1 word assigned to var "me"

ex: set me {John Hass} => same as above.

string operations: manipulate strings. Lots of such cmds available. Listing a few below. they all need to have keyword "string" before using the operation cmd

compare: string compare string1 string1 => returns 0 if string1 equals string2.

ex: if {[string compare $s2 $s3] == 0} { puts "Strings same"; }

equal: string equal $s1 $s2 => returns 1 if both strings are same

map: string map {abc 1 ab 2 a 3 1 0} 1abcaababc => replaces substring in string "1abcaababc" with key value pair. So, abc is replaced by 1, ab is replaced by 2 and so on. Result is 01321.

index: various cmds to get index of string
ex: string first string_to_find string1 => finds first occurreence of string "first string_to_find" in string "string1". Similarly we have "last" cmd to find last occurrence of string.
ex: string index $s1 4 => finds character at index 4 of string $s1

ex: string length $string1 => finds length of $string1

subst: This command performs variable substitutions, command substitutions, and backslash substitutions on its string argument and returns the fully-substituted result. If any of the -nobackslashes, -nocommands, or -novariables are specified, then the corresponding substitutions are not performed. When it performs its substitutions, subst does not give any special treatment to double quotes or curly braces (except within command substitutions)

ex: set a 44; subst {xyz {$a}} => returns ``xyz {44}'', not ``xyz {$a}''

ex: set tmp1  [subst -nocommands {${var1}isprime/name/${var2}/A1}] => Here var1 and var2 are substituted by their values and var "tmp1" is set to the resulting string obtained

format: format cmd formats string in fashion similar to C sprint cmd using % conversion specifier.

  • %d => convert integer to signed decimal string,
  • %f => Convert number to signed decimal string of the form xx.yyy. If we have %5.2f, it means 2 decimal precision with minimum field width of number (including all the digits and the dot) to be 5. It will pad with spaces to make it 5 wide if the total width turns out to be less than 5. ex: format %6.2f 78.345 => prints " 78.34". Note the space added in the front to make it 6 field wide. If we want to pad with "0" instead of space, we should do %06.2f, then it outputs "078.34".

ex: format " Today, %d shares in %s were bought at $%.2f each" 123 "Global BigCorp" 19.372 => Prints: "Today, 123 shares in Global BigCorp were bought at $19.37 each". Here %0.2f means 2 decimal precision.

2. list: This is a composite data type. It's a group of elements, put within quotes " ... " or curly braces { ... }. list is an ordered collection of items. List can contain any kind of items, even other lists.

List created using any of below:

ex: set myVariable {red green blue} => assigns list to var. Could have used " " instead of {}

puts $myVariable => lists can be printed by providing list name, prints "red green blue"

ex: list a b => returns {a b} or "a b" => other way to create list is to use list cmd explicitly. So, set myvar [list a b] is same as set myvar "a b"

IMP: Be careful when using {} when making a list. If there are var, then it won't work, and we need to use "" if we want to subs those var. ex:

  • set tmp1 "aca"; set tmp2 "baa";
  • set mylist [list {a $tmp1} {b $tmp2}] => Here {a $tmp1} is assigned to index 0, {b $tmp2} to index 1. No subs done.
  • set mylist [list [list a $tmp1} {b $tmp2}] => Here {a aca} is assigned to index 0, {b baa} to index 1. Subs done. Correct way if subs is desired.

list operations: Many operations can be performed on lists, as sorting, getting nth element of list, etc. There are many list cmds available that can do that. list is most widely used dat type in Tcl, so knowing cmds that do operations on list is very helpful. Below are few imp ones:

lappend => appends to list

set var1 orange
lappend var1 " " "blue" => appends blue and empty var "" to var1. Empty var does nothing to var1. instead of lappend, we can use append.

llength => length of string (if 2 elements in string, this returns 2)

puts [llength $var] => gives length as 2

lindex {list} index => returns element from list corresponding to said index. list's first element is index 0.

puts [lindex $myVariable 2] => prints index[2] which is "blue"
puts $myVariable => prints all elements of list


lindex {a b c} 0 -> a
lindex {a b c} 2 -> c
lindex {a b c} end -> c
lindex {a b c} end-1 -> b
lindex {a b c} -> a b c (no index provided, so the whole list is returned)
lindex {a b c} {} -> a b c

lindex {a {b {c d} e} f {g h} i} => This list has lists within the list (nested list). Here index 0 is "a", index 1 is "b {c d} e", index 2 is "f", index 3 is "g h" and so on. to access sub-elements of index 1 , we need to assign index 1 to a var, and access each element using lindex for that list, i.e set tmp [lindex { .... } 1]; lindex $tmp 2 => returns index 2 of {b {c d} e} which is "e".
lindex {{a b c} {d e f} {g h i}} 2 1 -> h (2 dim list, so equiv to lindex (g h i} 1 => h

lassign => xform list to var:
set var {orange blue red green}
lassign $var colour1 colour2
puts $colour1 => color1 gets assigned orange, while color2 gets blue

lsort => sort a list. default is ASCII sorting, unless options -integer, -real, -dictionary is used. ASCII sorting is basically sorting based on ascii value of each character.
set var1 [lsort $var] => var1 is ASCII sorted list (in increasing order, i.e smallest items first). -increasing sorts in increasing order (default), while -decreasing sorts in decreasing order.

lsearch {list} search_term => lsearch returns the index of the first element in list that that matches pattern, or -1 if there are no matches. Before Tcl 8.5 introduced the in operator, lsearch was very frequently used to test for the existence of a value in a list.

ex: if {[lsearch $var att] > -1} {

...

}

3. Simple array: This is a another composite data type like list. Just like var, no need to define array. Here array indices are integers like 0,1,2, etc.

ex: array_name (array_index) => NOTE: round brackets used instead of square brackets (square brackets are used in C pgm). In Tcl, square brackets used for cmd substitution.

set color(0) purple

set color(1) red

puts "$color(0) $color(1)" => prints purple red. If we don't put "" then it errors out since puts can take only 1 arg, while here there are 2 args

puts [array size color] => "size" returns the size of array "color". Here it returns 2.

To read elements of array, we can iterate thru indices using for loop (in case they are continuous), loop ending when we reach the size of array.

4. Associative array => associative array is a more genric form (or a superset) of simple array, where array indices can be anything (doesn't have to be limited to numbers). All scripting languages support associative array as it's very helpful. It's in key(index) value pair. Index doesn't have to be a number like 0, 1, 2, etc as in simple array. When index is string, it's called associative array. index is number in C like language, but in tcl or other scripting languages that support associative arrays, it can be anything.
ex: set marks(english) 80
array set marks english 80 => same as above. NOTE: no ( .. ) when using array set.

array set val [list a 1 c 6 d 3] => multiple var set here, by having more than 1 pair. So, array val(a) set to 1, val(c) set to 6 and so on. list is optional

array set val {a 1 c 6 d 3} => equally valid form, same as above. NOTE: curly braces used instead of [ .. ] since list is a cmd that returns o/p.

array set val [list a [list clk [list c d] e] z [list x y]] => here array elements themselves are nested lists, so val(a) = "clk [c d] e" val(z) = "x y". NOTE: these are not treated as nested arrays or 2D arrays or anything, it's still 1D associative array, just that each element of array is nested list. We can access each sub-element of val(a) by using list cmd as "lindex" etc. i.e "lindex $val(a) 1" gives "c d"


puts $marks(english) => prints 80, puts $marks is an ERROR as all indexes of array can't be printed directly, each index has to called separately
puts [array size marks] => returns 1 as size of this array

set marks(student) "Dave" => here array marks stores "Dave" and not number. Each element of array can store anything (obvious as everything is a string)

puts [array get marks] => "array set" and "array get" are most popular for setting array and getting array. " array get" returns a list containing pairs of elements (index and value) of the array.

ex: we can get all array elements by using [array get ...] cmd above, but we can also traverse using foreach loop.

foreach {subject number} [array get marks] { puts "$subject -> $number" } => here we get index and value in each iteration of loop.

puts [array names marks] => "names" prints indices of array. here it prints "english" since that's the only index. This is used to iterate thru indices of associative array using for loop, compared to using "size" to iterate thru regular arrays. "array get" could also be used to get all elements of array directly or going thru a loop.

ex: foreach {idx_key} [array names marks] { puts "$marks($idx_key)" } => prints value of each element of array

puts [array names marks english] => here it prints the index english, if it exists in marks array. since $marks(english) exists, it prints "english". If english was not a valid index, it would have printed nothing.

ex: index can be hyphen or anything in combination of special char.

set results(-noexecute) false

set Me $results(-noexecute) => sets Me to "false"


operators:

arithmetic: + - * / %
relational: == != < > <= >=
logical: && || !
bitwise: & | ^ << >>
Ternary: ?: => If cond=true ? val=x : val=y
math func: abs, sin, cos, log, rand, etc
system func: clock, open, close, exec(execute system cmd)
ex: puts [exec ls] => executes ls cmd, and output is printed on screen

NOTE: there is no "=" operator in tcl as in C or other languages. To assign something as in tcl, we use "set" cmd explained next.


Cmds:

1. set => sets a vraiable. In most other languages "=" sign is used to assign a var, but in tcl, set is used instaed (there's no equiv "=" sign in tcl)
----
set a 5 => sets var a to 5. Use $a to access a. puts $a will print 5. here 5 is string type
set a "me" or set a {me} or set a me => sets var a to string me. NOTE: "set a me you" won't work as "me you" is 2 words and not interpreted as 1 string, unless put in quotes => i.e set a "me you"
set colorlist {red green} => list is group of words within {} or "". $colorlist lists all the elements
set colorlist1 [list red green] => same result as above, list is optional
set colorList3 [split "red_green_blue" _] => the cmd split, splits the colors and then list is set those 3 colors. => puts $colorList3 => red green blue

2.

3. expr: In tcl, we can't directly do any arithmetic, as everything (incl numbers) is treated as string, so string operation is done.Instead we use cmd "expr" or expression to indicate it's an arithmetic expression.

ex: set a 2; set b 5; set c $a+$b => sets c to 2+5 and NOT 7. To fix this we need sto do this: set c [expr $a+$b] => sets c to 7. 

4. exec:

5. regexp:

regular expressions: pattern match similar to perl can be done here. regexp is used everywhere in unix world. See the regular expression page. tcl regexp is consistent with linux regex standard. However, here regexp has to be inside { }. This prevents var expansion (i.e $a will be treated as literal $a without any expansion for var a). If you want expansion of var, then you need to use double quotes for pattern "pattern". Also regexp supported seems to be ERE and not BRE. So, () and {} are treated as metacharacters and NOT literals.

NOTE: Tclk 8.1 and above has moved to advanced regex, so the behaviour for tcl versions before and after 8.1 may be different. See this link for tcl regex: https://wiki.tcl-lang.org/page/Regular+Expressions

NOTE: When you move from {} to "" for patterns, make sure that additional escape chars are added. For e.g., if you had {\s*}, change it to "\\s*". I don't know the reason. Double quotes have unexplained behaviour, so I stick with {...} in patterns, as they follow what you would expect of regex behaviour.

  • ex:  regexp {.*\.\*$} $mytmp => This matches .* at the end of string. If we replace {..} with "..", then it matches anything as \.\* are interpreted as .*.*. However to match {...} behaviour, we have to put \\ before . and before *, i.e regexp ".*\\.\\*$" $mytmp => This matches the bahivour of {...} where .* is matched at the end of any string.

syntax: regexp optionalSwitches patterns searchString fullMatch subMatch1 ... subMatchn => patterns are searched for in searchString. Full match is any variable to hold the result of matched regex result. Submatch1 to SubMatchn are optional subMatch variable that holds the result of sub match patterns. These submatch happen if we have () inside the pattern, which implies that we want to hold the result of this match in a separate var.

return value of a regexp is 1 if matched, and 0 if it didn't match. So, this can be used in an if-else condition (NOTE: put [] around regexp to pass the return value)

  • regexp abc abcd => Simplest regex cmd, where "abc" is pattern searched in string "abcd". Here, it matches, so it returns 1.
    • if {[regexp abc ab]} { ... } => This doesn't match (as pattern abc isn't in string "ab"), so if part not executed.
  • regexp {([A-Z,a-z]*)} "Tcl Tutorial" a b => regexp is cmd that searches for string with any alphabets in the given string "TclTutorial". Any pattern matched string is assigned to var a. Since there is () inside the pattern, other match is also done in exactly same way, but the result is now stored in another var b. This returns a=Tcl, b=Tcl.
  • regexp {([A-Za-z]*.([A-Za-z]*))} "Tcl Tutorial" a b c => Here a is matched to the whole thing, with b also matched to whole string. c is matched to string within b as sub pattern (A-Za-z]*) is inside ([A-Za-z]*.([A-Za-z]*)). So, a=Tcl Tutorial, b=Tcl Tutorial, c=Tutorial
  • regexp {[0-9]+[ ]+:[ ]/} "12: 34 : 1234 : /My name" => this regexp matches 1 or more digits followed by 1 or more whitespace, then single : followed by 1 whitespace, ultimately followed by /. So here it matches "1234 : /". NOTE: whitespace match by giving a space inside square brackets. i.e: [space]. To match any of newline, tab or space do:  [\n\t ]+

switches: There are multiple switches or options that can be applied to regex via -. A "--" (double dash) implies end of switches.

  • -nocase: ignore case
  • -line: matching done only on a line, ignoring characters after a newline.
  • -start: Sets the offset of start of search pattern. So, -start 4 starts searching after 4th char (i.e 5th char onwards)
  • -- end of switches.

ex: regexp -nocase -line -- {([A-Z]*.([A-Z]*))} "Tcl \nTutorial" a b=> Here a and b, both match "Tcl" as newline prevents the regexp to go to next line (-line option)

regsub: Just like regexp is used for seraching matches, regsub can be used to substitute regex matches in a string.

Syntax: regsub opt_switches regexp subject replacement opt_resultvar => If opt_resultvar not specified then substituted string is apssed to the result of this cmd (which may be used to set another var, etc). -all switch matches and replaces all instances of patterns in the string. Without this switch only the first matching range is found and substituted.

  • ex: regsub -all {\mfoo\M} $mystr bar my_str => Replace (in the string in variable mystr) every instance of foo which is a word by itself with bar. Stores the final result in the same string my_str (NOTE: no $ in resultvar)
  • ex: regsub -all {.tcl} $my_list "" my_list => remove .tcl in $my_list var.

6. io cmds: cmds for input output of data. Just like scanf and printf in C. We use puts for printing, and gets for getting input. o/p cmd "echo" doesn't work in tcl, unless we do exec (i.e exec echo "abc"). However, echo does work when tcl files having "echo" cmds are sourced into CAD tools. I've many examples that use "echo" instead of "puts". That's not correct, and may not work. Use "puts" instead when working in tcl, as that's the proper print stmt.

  • A. puts: To output to screen or other files
    • puts -nonewline string => writes characters given by string on channelId which by default is stdout. puts normally outputs a newline character after string, but this feature may be suppressed by specifying the -nonewline switch.
      • ex: puts "my name is $name" => This prints "my name is Raj", assuming var "name" is set to Raj.
      • formatted printing: Similar to printf in C, we can formt o/p by using format cmd in tcl, which formats any string. See format cmd abobe in strings
        • ex: puts [format "%0.2f" $rate] => string is first formatted and then passed to puts cmd.
      • ex: Append a log message to a file:
        • set chan [open my.log a] => nested cmd open inside []. chan is set to filehandle returned by opening my.log in append mode (a=append, r=read, w=write)
        • set timestamp [clock format [clock seconds]]
        • puts $chan "$timestamp - Hello, World!" => since file is in append mode, this line is written at the end.
        • set file_data [read $chan] => reads file and stores contents in file_data. File has to be open in read mode.
        • puts $file_data => prints all contents of file_data on screen
        • gets $file_data data1 => gets data from file one line at time, and puts in var data1. use a loop to read whole file
        • close $chan
  • B. gets: To get input from screen or other files.
    • ex: Get a number from user and print it
      • puts "Enter a number
      • gets stdin my_var=> gets the input from std input (keyboard) and stores it into var named "my_var"
      • puts "Num entered is $my_var"
  • C. redirect => to redirect o/p of cmd to a file. We provide the name of the file (and NOT the pointer of file when doing "open file"). Reason is redirect will open and close the file for you, instead of you trying to open a file in wrt mode, write into the file and then close the file. This becomes very convenient as it's all done under the hood with redirect cmd. Do NOT use file pointer with redirect cmd. It will not write anything, nor will it show any error.
    • ex: redirect -file ~/tmp_file {[puts $marks(Mark)]} => -file specifies file_name to redirect the o/p to. Anything within { ... } is redirect to the file. "puts" is needed since it prints the o/p of cmd. Std unix redirection cmd ">" will not work for redirection.

8. flow ctl cmds: controls flow, via loops, conditional stmt, etc. Similar kind of keywords as in C: if-else, while, for, etc. Syntax for these is very stricyt in tcl, where starting/closing braes have to be in

A. if - else: You have to add if else start and end curly braces on same line (as shown below), except for the  final closing curly braces }. If we put starting or ending curly braces on separate line else it will error out with "unkown cmd {".
if {$x == 0} { => IMP: if starting curly braces should be on same line, exactly like this. You can't write if {$x == 0} on 1st line and "{" on next line, as then the cmd if won't be parsed correctly. gives error "wrong #of artgs" for if.
echo "Equal"
} elseif {$x > 0} { => IMP: write elseif exactly like this. ending "}" for if and starting "{" for elseif need to be on same line, so that elseif cmd can be recognized correctly
echo "Greater"
} else { => IMP: write else exactly like this. similar to elseif above
echo "Less"
} => This could also be put in previous line.

ex: if {$x == 0} {echo "Equal"} elseif {$x > 0} {echo "Greater"} else {echo "Less"} => This is same as above cmd, but everything on 1 line. This is equally valid, as all cmds can be parsed correctly here (since there are matching starting closing braces around keywords)

B :? => used to replace if else
set b [expr $a == 1 ? 20: 30] => assuming a=5, then, b is set to 30
#while
set p 0
while {$p <= 10} {
echo "$p squared is: [expr $p * $p]"; incr p
}

C. for: rewriting while loop shown above
for {set p 0} {$p <= 10} {incr p} {
echo "$p squared is: [expr $p * $p]"
}

D. foreach: implements a loop where the loop variable(s) take on values from one or more lists.
#simple list
foreach el [lsort [array names a]] { => each array name a is assigned to el
echo "a\($el\) = $a($el)"
}
#multiple items from list
foreach {name gender} [list Bob male Sarah female Kim hermaphrodite] {
... => name and gender assigned to "Bob and male" in 1st iter, then "Sarah and female" in 2nd iter, so on. $name and $gender can now be used inside the loop.
}
#multiple lists traversed simultaneously
foreach a [list 1 2 3 4] b [list 5 6 7 8] c [list a b c d] {
puts "$a $b $c" => "1,5,a" is printed in 1st pass, then "2,6,b" and so on.
}

E. break, continue: terminate a loop before the termination condition is reached, as follows:
The break command causes the innermost loop to terminate.
The continue command causes the current iteration of the innermost loop to terminate.

F. switch: equiv to if tree
switch $x {
a {incr t1}
b {incr t2}
c {incr t3}
}

9. tcl special cmds: These are special cmds, which are used very often inscripts, so I've listed few of them.
A. info script:
echo "Sourcing [ file normalize [ info script ] ]" => info script returns name of file containing the script that is currently being evaluated. file normalize file1 => returns normalized filename (no ../ etc) for name1. This echo line should be first line in any script to print name of file/script being run. For some reason, this gives an error with "echo is unknown cmd". Use puts [exec echo "..."] instead. Prints => Sourcing /home/kagrawal/scripts/hello.tcl

B. info exists:
[info exists ::env(VAR1)] => returns 1 if such env var VAR1 exists, and is defined, else returns 0.
set a 3;
info exists a => returns 1
info exists $a => returns 0, since $a is 3, so it looks for var named 3, which doesn't exist.

10. procedures: Similar to functions in C.

#procedures: reusable procedures can be written. proc name {args} {body}

var d; set glb_d "my name";
proc plus {a b} { variable c; upvar $a arg1; return [expr $a + $b] } => 1 line proc defn to add two numbers. expr=expression. Here variable "c" is only seen within plus namespace (see namespace section below). To see it globally outside the proc, there are 2 ways:

  1. global: Decalraing a var as global inside a proc, allows it to be seen outside the proc.Same "global" keyword is also used to make a var declared outside the proc to be visible inside the proc.
    • In above ex, use "global c" inside the "plus" proc. Then var c is accessible outside the "plus" proc
    • In above ex, use "global glb_d" inside the proc. Then var "glb_d" is accessible only inside the proc "plus". If we have "global glb_d" outside the proc, then it will have no effect, and won't be accessible inside the proc.
  2. upvar: upvar creates link to variable in a different stack frame. It simulates "pass by reference" seen in other languages as C. This is able to handle complex situations which global can't handle.
    • Syntax is upvar <level> <other_var> <my_var>. <my_var> can access <other_var> from corresponding stack frame. Default <level> is 1. We can provide <level> as number 0, 1,2, etc or #0, #1, #2, etc. When # used, the meaning changes slightly. "upvar #0" means at global level (i.e get values from top level). In above ex of proc, upvar makes arg1 an alias for var a, so that arg1 can be used inside proc w/o issues. Generally convenient to do it this way. Only use #0 or #1 (as per tcl doc) or you are looking for trouble.
    • In above ex: use "upvar #0 glb_d loc_d" inside the proc. Then "loc_d" var link is created for "glb_d". Now if we modify glb_d or loc_d, both var get modified as they are refrring to same mem location. Ex:if inside the proc, we do => set loc_d "you". then both glb_d and loc_d are set to "you"


puts [plus 5 6] => proc being called with args. returns 11. NOTE: no brackets should be used for args when calling proc, or else it will give "Error: wrong # args". Also proc called has to be put into [...] when it's being used as arg to other cmd. Else "plus" keyword will be treated as data, and will error out. if proc is used by itself on a line then we can write "plus 2 3" (without the quotes) and it will work fine. However, if we use [plus 2 3] by itself on a line, then we get [5] (with the sq brackets) which is not what we want.

 

define_proc_attributes => Defines attributes of a Tcl procedure,
ex: define_proc_attributes sta_flow -info "configure pt" -define_args { => -info "text" gets printed when help is typed for this proc, help -verbose prints all possible args below which is a list of lists. Can be just 1 arg too.
{-design "valid design name; NOTE: usually defined by project.globals" "" string optional} => this is arg1
{-stage "valid design stage; NOTE: usually defined by project.globals" "" string optional} => This is arg2 and so on ..
}

ex: factorial
proc factorial {number} {
if {$number <=1} {return 1}
else return [expr $number * [factorial [expr $number - 1]]]
}
puts "factorial of 5 is" factorial 5 => prints 120

ex: info body factorial => This returns the code in the proc "factorial". This is helpful during debug, so that we don't have to find where that proc is.

11. namespace: Namespace is a container for set of identifiers that is used to group variables and procedures. Namespaces are available from Tcl version 8.0. Before the introduction of the namespaces, there was single global scope. Now with namespaces, we have additional partitions of global scope.
Namespaces are created as below:

namespace eval MyMath { => creating namespace MyMath
variable myResult => var inside namespace, can be accessed via MyMath::myResult
proc Add {a b} { .... } => This proc Add is only within MyMath namespace
namespace eval name2 { ... variable myResult ...} => namespace can be nested too. ref would be ::Mymath::name2::myResult
}
proc MyMath::Add {a b } { => create proc "Add" inside MyMath namespace, need to explicitly use ::, since this proc is defined outside MyMath amespace { ... }
set ::MyMath::myResult [expr $a + $b] => assign MyMath var MyResult to result, :: refers to top level global namespace, and then further :: refer to lower level. We can omit global ::
}
MyMath::Add 10 23 => this sets MyMath::myResult to 33
puts $::MyMath::myResult => prints 33, we can omit global ::, i.e puts $MyMath::myResult

To avoid using lengthy scope resolution operator, we can import/export namespace
namespace eval MyMath {
variable myResult
namespace export Add => This says that make "Add" available to whoever imports MyMath. Using export/import allows names to be shortened
namespace export Add1 => export another proc Add1, do similarly for other proc that you want exported
}
proc MyMath::Add {a b } {
return [expr $a + $b]
}
namespace import MyMath::* => Here everything that has been exported, is imported in global namespace
puts [Add 10 30] => Since "Add" has been imported, it is available in global namespace, Var still available as $MyMath::myResult

12: packages: similar concept as in other pgm languages, where a group of unctions, header files, etc can be put in a set of files, which can be acccessed as a single entity.

creating packages => consists of collection of files (tcl scripts, binary, etc) that provide specific functionality. It has a pkg name. Once created, pkg can be called in other tcl pgms, by specifying path of pkg
To create a package with TCL, you need two files :
  1.  the package file which contain your code
  2. the index package file to declare your package

These are the stpes to create pkg and then use it in a pgm:

I. package file: create a tcl file called HelloPkg.tcl. This defines the pkg called "HelloP" with all procs and var in it. NOTE: name of pkg file (HelloPkg.tcl) can be different than name of pkg (HelloP) as well as the namespace "HelloWorld". Strictly speaking, namespace is not required, as we just load this file, so procs and var will be available to the whole pgm loading this file. However, convention is to have a namespace so that if multiple pkg are being loaded, and they happen to have same name of proc or var, then they will keep overwriting each other, creating errors in results. Also, we usually keep namespace and pkg name the same, so that it's easier to know from which pkg, the particular proc is coming from.

namespace eval ::HelloWorld {
        namespace export Talk => export proc Talk, so that it's available to anyone who uses "import HelloWorld::Talk"
        set version 1.0
        set HelloWorldDescription "HelloWorld"
        variable home [file join [pwd] [file dirname [info script]]] => Variable for the path of the script
}

proc ::HelloWorld::Talk {} { => define proc Talk
    puts $HelloWorld::HelloWorldDescription
}
proc ::HelloWorld::GetPath {} { => define proc GetPath
    variable home
        return $home
}
package provide HelloP $HelloWorld::version => "package provide" actually creates pkg. NOTE: we defined name of pkg as "HelloP" while namespace is "HelloWorld"

package require Tcl      8.0 => This is needed incase Tcl version is < 8 (then namespace won't work, as they are defined only for version 8 and higher)

II. index package file: This index file is simply a Tcl file which is loaded into the interpreter when Tcl searches for packages. This file is named pkgIndex.tcl.  pkgIndex.tcl can be written manually or can be autogenerated. We need to add this 1 line in pkgIndex.tcl:

package ifneeded HelloP 1.0 [list source [file join $dir HelloPkg.tcl]] => "package if needed <pkg_name> <version> <pkg_filename>" cmd loads this pkg (pkg_filename) in response to cmd "package require <pkg_name> <version>". pkgIndex.tcl file is sourced on application startup, but pkg files are not loaded at that time since "ifneeded" is used. If "if needed" is not used, then <pkg_filename> will be loaded while sourcing of pkgIndex.tcl, which may be unnecessary, as not all packages are used in every pgm. "ifneeded" prevents pkg from getting loaded. The second part of this cmd says what files to source when this pkg is going to be loaded. We need full path of pkg file for sourcing to happen. $dir is the path of dir where HelloPkg.tcl resides, so "file join" provides full path of file. NOTE: $dir is not a std var, and "echo $dir" returns "unknown var" error, but it works for some reason when used in pkgIndex.tcl.

package ifneeded HelloP 1.0 [list source [file join $dir hello1.tcl]]\n[list source [file join $dir hello2.tcl]]\n[list source [file join $dir hello3.tcl]] => We can load multiple files too for 1 pkg

To auto generate this file, we can cd to dir where our pkg file "HelloPkg.tcl" is located, open up tclsh, and run "pkg_mkIndex . *.tcl" cmd. This will generate " pkgIndex.tcl" file in that dir for that pkg. If there are multiple packages in that dir, then pkgIndex.tcl will have multiple lines, 1 for each package.

III. use package in external pgm:  Tcl searches for index pkg file in "auto_path" or "tcl_pkgPath" which store default paths to search.

echo $auto_path => /usr/local/lib /usr/share/tcl8.5 /usr/lib64/tcl8.5 /usr/lib64/tk8.5 /usr/share/tk8.5

echo $tcl_pkgPath => /usr/lib64/tcl8.5 /usr/share/tcl8.5 /usr/lib64/tk8.5 /usr/share/tk8.5

Since we may not want to put our local file in any of these standard /usr/ dir, we append any of these paths to add the path where our pkgIndex.tcl is going to be put. Now to use this pkg in any tcl pgm, add these 3 lines in the pgm, Hello.tcl

lappend auto_path "/home//scripts/test" => This is the name of dir where pkgIndex.tcl is located, we append this path to the search path
package require HelloP 1.0 => "package require" cmd loads pkg. here pkg "HelloP" version 1.0 is loaded.
puts [HelloWorld::Talk] => Once pkg loaded, all proc and var accessible via namesapce.

puts [HelloWorld::GetPath]

puts $HelloWorld:::HelloWorldDescription => To access a var in pkg, we need to precede it with $

On running the script, Hello.tcl, we get this o/p:

HelloWorld
/home/scripts/test
HelloWorld

12. Error cmds: Error handling in Tcl is provided with the help of error and catch commands. Basically it allows you to take a chosen action incase any cmd in script gives an error. This is useful for debug purpose in production.

syntax:

error message info code => message=error msg, info=error info stored in global var "errorInfo". code=error code stored in global var "errorCode"

catch script_or_cmd ErrorVar => script_or_cmd is any cmd that you normally execute in your tcl script, ErrorVar is the name of var used to store the "Error msg" that is provided in the error cmd above. Catch cmd returns 1 incase of error, else 0. Useful to use it within if stmt, so that we print some debug info incase of error.

Ex:

proc Div {a b} {
   if {$b == 0} {
      error "Error generated by error" "Info String for error" 401 => Error var stored in global
   } else {
      return [expr $a/$b]
   }
}
puts "Result = [Div 10 0]" => Here we are executing Div proc normally. Since we provided 0 as one of it's operand, it will print error msg from above
if {[catch {puts "Result = [Div 10 0]"} errmsg]} { => Here, we catch that errorinfo from above, and take a defined path on printing these errorvar that were stored above in proc. This is more user friendly and also easier to debug.
   puts "ErrorMsg: $errmsg"
   puts "ErrorCode: $errorCode"
   puts "ErrorInfo:\n$errorInfo\n"

}

catch "exec myscript $a 4b" => this will catch the error, bt is not doing anything with it. This also doesn't have a var to store error msg. This is not very helful use of catch. We can use catch cmd with all other cmds, but then script will be very long and unreadable. So, we should use catch only with cmds that seem likely to fail, like "opening a file", etc.

13.

 



Tk = ToolKit

Tk provides cross platform gui widgets. It can be used with other dynamic lang, not just Tcl

Hello.tcl
---
#!/usr/bin/wish => wish interpreter is needed to run Tk cmds, if we use /usr/bin/tclsh, thne it will not run as any Tk cmds are not identified by tclsh
grid [ttk::button .mybutton -text "Hello World"]
---

./hello.tcl => draws a box with "Hello World" in it. We can also type > wish hello.tcl => not needed since wish interpretor is already there on 1st line

special var:tk_version, tk_library

#!/usr/bin/wish
puts $tk_library => prints path of where all std Tk libraries are. Note: all tcl cmds are still valid in Tk

widgets: basic component pf Tk based app. aslo called window
-------
main widget is root widget, and other widgets can be placed in it. basic gui widgets are buttons, menus, etc.

create widget:
type variableName arguments options => type= widget type as button,label, etc. args are optional. options range from size to formatting of each component

ex: label .myLabel -background red -text "Hello World" => root window is named with . and an element in window is named .button1. button1 is var name. . refers that it's in root window

basic widgets:
-----------
label = Widget for displaying single line of text.
button = Widget that is clickable and triggers an action
entry => accepts single line of text as i/p
message = Widget for displaying multiple lines of text
text = Widget for displaying and optionally edit multiple lines of text.
toplevel = Window with all borders and decorations provided by the Window manager.

layout widgets => frame, place, pack, grid (widget to nest widgets packing in diff dirn)

selection widgets

media widgets

ex:
grid [label .myLabel -text "Label Widget" -textvariable labelText]
grid [text .myText -width 20 -height 5] .myText insert 1.0 "Text\nWidget\n"
grid [entry .myEntry -text "Entry Widget"]
grid [radiobutton .gender.maleBtn -text "Male" -variable gender -value "Male"
-command "set myLabel1 Male"] -row 1 -column 2
canvas .myCanvas -background red -width 200 -height 200
pack .myCanvas
.myCanvas create arc 10 10 50 50 -fill yellow

window maager => to handle top level window. . refers to main window
wm option window arguments

ex:
toplevel .t => creates top level window with name "t"
wm maxsize . 800 800 => max size of toplevel window is 800 by 800. Can't be extended by more than that by stretching with mouse
wm minsize . 300 300 => min size is 300 by 300
wm title . "Hello" => title displayed on window is "Hello"
wm attributes . -alpha ".90" =>
wm geometry . 300x200+100+150 => width=300, height=200, x position on screen=100, y position=150. This position determines where on your monitor screen the window is going to show up.
grid [button .mybutton -text "Hello \n World"] => places button on 1st line of grid


 

joomla:

Joomla is very popular open source Content Management Software (CMS) right behind Wordpress in popularity. It's little bit more complex than Wordpress, but has lot more options to customize your website. It's not written around making a blogging website, but to make a regular website. With wordpress too, you can make a regular website, but it will require more work. In joomla, you can start right from the box.

Joomla offical website is: https://www.joomla.org/

A lot of begineer's and advanced learning material is on: https://docs.joomla.org/Main_Page

A very good tutorial as always is on tutorialspoint website: https://www.tutorialspoint.com/joomla/

Joomla 3.9 is latest as of June, 2020. All material below is wrt version 3.9.

Joomla vs WordPress:

Before you install joomla, understand that there are some very basic things missing from joomla that are very standard in Wordpress. I hate joomla for that, but then the lure of having a "customized interface" in joomla made me switch (customization in wordpress is too much work, as it's mostly for blogging. However, there are too many templates available in wordpress, that can allow you to customize any website). Joomla has lots of advanced features, which most of the times are just more fancy and convenient, but don't really add any real value. In fact things of real value have been taken out of joomla, as I note some of them below.

1. Joomla does not have auto save feature. So, basically if you have been typing an article for an hour, and then suddenly realize you have to save, and then you click on save. But then something goes bad (internet connection lost, timeout, autologout etc). Now the page you were trying to save with latest content is lost forever. You still have access to older copy, but your work for last 1 hour is gone. It's not saved anywhere. Wordpress auto saves your articles every few minutes, so I've never lost any article there, but with joomla I've lost lots of articles. One option is to click on "toggle editor" button on bottom of editor, then do select all and copy (ctrl+a, ctrl+c), then paste (ctrl+v) everything in local editor on your local machine, and save it there. Then click on "save" button on joomla, and if for some reason, save doesn't succeed, you have a local copy, from where you can copy and paste again. NOTE: you should always save "html code", as saving raw text loses all markups as bold letters, underline, etc. Also, be default the session timeout is 5 minutes, which is too small and on doing save, it's guaranteed that it will log you out without saving your new content. I extended it to "couple of days", so now I don't lose my articles due to auto log out (better than nothing). This can be done by logging into administrator panel, and clcking on system->global configuration. Now select "system" tab on slight right on top, and scroll to bottom on "session lifetime". Set it to 15000 and save.

2. Joomla doesn't have any search feature to search for articles in your backend (administrator panel). In wordpress, you can search for anything in all articles in backend. This was very convenient, as many times I would not remember which article I put certain information in. Searching for this easily allowed me to find the article. In joomla, there is a search button in backend, but it only searches in title, which is not very helpful. For ex, if I'm looking for a term "TV show" in any of the articles, searching in backend will report only those articles that have this term in the title of the article. The only way to search is to make all articles "published", then goto frontend (i.e your website "xyz.com"), and then search over there. There it searches for that term in all articles, menu items, etc.

3. mouse right click doesn't work in editor when editing articles in joomla. So, you have to goto "Edit" button at top and click on copy,paste etc. This is not too bad, as there are also short cut keys that you can use (ctrl+c to copy, etc0

intsalling joomla:

I've joomla 3.9.10 installed. So, instructions below may differ for other joomla versions. Official joomla website is best resource for installing joomla: https://docs.joomla.org/J3.x:Installing_Joomla

Few tips:

  • Download Filezilla client to upload joomla files to your server (filezilla is ftp pgm to transfer files from 1 computer to other)
  • On some host, "localhost" doesn't work for hostname in database configuration. Get name of server from you web hosting provider. Ex: on ipage, it's <yourloginname>.ipagemysql.com (i.e if your login for ipage is matt2016, then hostname would be matt2016.ipagemysql.com), while on godaddy it's localhost.
  • Once successfully done, you may want to remove login panel from front page of your new website. Goto Extensions->Modules, then search for "login", and then where login form appears, "unpublish" it. That will get rid of the login panel.
  • To go to administrator site (aka backend of website), type your website name, and add "/administrator at end. ex: www.maaldaar.com/administrator/

joomla basics:

We'll log into adminstrator site (backend), and then look at various buttons on top. These are the icons you will see on top: System, Users, Menus, Content, Components, Extensions and Help. Most of the things under Users, Menus, Content and Extensions can also be accessed by clicking on System->Control panel.

Template: Templates are what determine how our website looks. It's a type of joomla extension, and you can download any joomla template on internet, and activate it on your website. That will completely change the look of your website. Thus templates make your website customizable. If we click on "Extensions-> templates" on top, we'll see 2 templates available by default (Beez3, protostar). We can also also see the 2 templates by going to system->control_panel->templates. Under styles, it will show all templates. Choose default icon for the template you want to have for your website (by clicking on default). protostar is default, but personally I prefer Beez3, so I change it to Beez3. All website creation notes below have been written with Beez3 in mind. The templates that we talked above are front end template. There are also 2 back end template that change the feel of the administrator panel (after you log in). You can see these by choosing "administrator" instead of "site" on left top side, on the screen where you see the 2 front end templates. The 2 back end templates available by default are Hathor and Isis. Isis is enabled by default. We rarely need to change it, so we never mess with back end template. 

Each template comprises a lot of files. Main file is index.php, which is called on loading your website. This in turn calls many other files which determine the look and feel of your site. You may change the code in the any of the files in this template, and that will change the website appearance. Thus templates contain the main code of your website, and they interact with contents of your website in order to display it. Files in a template can be viewed by by clicking on extensions->templates->templates (not template style), and then clicking on particular template details and files. You will see index.php as well as various dir. Whenever you want to change anything in template, you will need to mess with these files, so you will need to know some php, css and html coding.

3 important concepts to know regarding publishing a page or an article:


1. ARTICLE: first create article using article manager. Articles are the main content that you want displayed on your website (similar to wordpress article concept). We can see all articles by going to Content->Articles. When we see all articles list, you should make sure that you see a green tick mark in "status" in front of any article name. Else that article will not display on your website. You can add a new article by clicking on "new". By default, "content" tab is opened, which gives us a blank editor to type our content. On right side are multiple options: choose "status" as published, Category as "what you want" and other options. There are multiple tabs on top besides the "content" tab. "publishing tab" is a important one, which controls the"status" of the article. If the "finish Publishing" line has a date on it, then the article will expire on listed date, and will cause the "status" to go red. Delete the "finish publishing" line to keep article unexpired. On saving the article, you will see that each article gets assigned an "ID" number. This is the unique number that is used on the link to display the article. For ex: to see article with ID=23, http://www.maaldaar.com/index.php?option=com_content&view=article&id=23. However, when you go to that article on your website, it will have a different link like: http://www.maaldaar.com/index.php/linux/linux-installation. This is because joomla creates an alias for that link, which is human friendly.


2. MENU: However, articles by themselves are not visible on your website, unless they are linked via "menu". Menu is list of menu items. You can think of menu as "restaurant menu", which has various menu items as bread, burger, chips, etc. In joomla, menu items contain the final link to your articles. You can view menu as rectangular box with hierarchy of menu items in it. Usually, it's "top/bottom or left/right navigation gui" box, which has all menu items in it. Clicking on menu item takes you to the page, which has one or more articles in it. Infact, menu item is just a link to whatever you assign it to (we usually assign menu item link to an article, but it can be link to many other things. Joomla allows us to choose what we want that menu item link to point to). For ex: menu might be "Shopping", while menu items might be "electronics", "clothes", "jewelry", etc. All of these menu items together form menu. Do not think of menu and menu item as same thing, they are NOT. menu is a collection of menu items.

If we click on "menu" on top, we will see 2 menu provided by default: main menu and user menu. The default is main menu (shown by home sign). We can add menu items to "main menu" by going to meus->main menu->Add new menu item. On new screen, we choose "menu item type" to be "article", link it to an existing article, save and close. Thus the article gets linked to menu item. We can add more menu items, each of which has link to articles, etc. Ideally there should be a separate menu item for each article, if you want each menu item to point to a article. Any article that doesn't have a link via menu item, will never appear on your website. We can also add new "menu" on top of 2 default menu provided. Click on menus->manage->add new menu. We can have as many menu as we want. Usually, we put all menu items under default "main menu", and that suffices for simple websites.

So, what's the purpose of having more than 1 menu? It's so that we can have different custom menu to display on different pages or on different parts of the same page. For rx, we may have a side menu on our website on left side, which always shows all menu items. However, on top row of website, we may want to have a menu with different menu items as "About Us", "contact", "sitemap", etc. In such a case, we need 2 different menu. We'll see how to place these menu in module section.

We can also create a hierarchy of menu items, so that our website is easy to navigate. When we create or edit menu item, on right side, we see some options. One of the options is "Parent item". Here, by default, it's selected as "Menu item root", but can be chosen as any menu item already created. Then that menu item becomes parent, implying that this menu item is child of that menu item, and sits under it. Similarly, we can have more child menu items. This is very useful to keep your website hierarchical, instead of flat.

An example of such hierarchical menu item is a website that has "finance", "shopping", "real estate" as menu items, and then each of these have further menu items (i.e finance has menu items as banks, stocks, bonds, etc, while shopping has menu items as beauty, clothes, etc). We can link an article to each of these. If we want parent menu items as "finance" to contain something different (i.e have a link of menu items that are under it, or short summary of menu items under it, we can choose "archived articles" etc in it instead of "single article". So, from perspective of joomla, all menu items, whether parent or child, are treated same way. T

Other way to understand difference between menu and menu items is that the top level root menu item is called "menu", while all child items underneath it are called menu items. Thus even though menu is just the top level or parent of menu item, we differentiate b/w menu and menu item, since menu is the one that is called by modules (explained below).


3. MODULE: The 3rd important thing is module (after article and menu). We created menu above (with menu items in it), but it's not visible on website yet. Where and how to place it is decided by module. Menu will not show up on webpage unless it is linked to atleast 1 module (and that module is published or visible). This is because, modules are the ultimate thing that show up on webpage. In template "Beez3", we see html/modules.php file (by going to editor of beez3 template. It's in extensions->templates->templates and clicking on details and files). This file is called via index.php and all modules are loaded and rendered one by one. So, if some menu is not linked from within a module, it will not show up on the website. We can see all available modules by clicking on extensions->modules. For main menu above, we have a default main menu module. Similarly we have "login form" module, "latest articles" module, etc. Position of each of these modules (i.e where they are going to be placed on the webpage) is indicated under "positions". Position of these modules can be changed to anywhere on your webpage as left/right/top/bottom etc. For Beez3 template, main menu module is in position 1 (top) by default, while user menu module is in position 7 (left). Which position goes where on the page is determined by the template code, i.e in template beez3, developer of this template might define "footer" position as one on the bottom of the page, while position 7 might be defined as right of the page. So, when arranging positions of various modules, choose module position corresponding to that template.

We can create many more modules for many different positioning of same main menu. Or we can create new menu as "my custom menu", and create a new module for "my custom menu". Thus module allows us to have flexibility in placement of that box, which may be menu, search form, etc. Without module, menu would be at a fixed location, with same menu items being seen across all articles. The code is hardcoded into the template. But now, we can make many menu (each having different or similar menu items), then make many modules (with each module having different menu). Now on each webpage, we can decide what module to place, where to place, so that different menu will show up on each webpage. This is a big winning feature of having module in joomla (in wordpress, you have same menu across all pages. modules is a new concept introduced in joomla). Thus modules provides one extra level of customization.

As you can imagine, module can be of many type. We can define module as "menu type", where it contains menu. Or it can be "article type", where it contains articles. Thus we can create a new module, assign it a type, and then choose what to display when that module gets loaded (i.e display menu items, articles, etc). Mostly, we have modules as "menu type", as we create menu items to link to our articles. For this website, I already had a "main menu" module, which was of menu type, and it's position was assigned 7 (i.e left). So, when the template gets called, this "main menu" module gets loaded, which is linked to "main menu". So, each module is a rectangular box and it contains inside it whatever we want it o have. All these modules then get displayed on any page at positions assigned, and this is how our final page looks !!

Modules are confusing concept, and the positioning is even more confusing since each template may have it's own definition of what "position-0" means. To help with that, joomla allows positions of modules to be seen on your website itself. For that, first goto Extensions->Templates, then click on "options" icon on top right of that page. Over there, set "Preview module position" as enabled, save and close. Now open your webpage (www.xyz.com), and append ?tp=1 (tp=1 says enable "template positioning") to end of that link (i.e www.xyz.com/index.php/shopping?tp=1). This will show position and style (i.e Position: position-0,  style: none outline) of all the modules that are rendered on that webpage. This is very helpful in understanding and debugging module positioning related issues.

Summary: So, to get your first webpage to be published, you need to understand all 4 things -> article, menu, module and template. Write a new article, then create menu/sub-menu which ultimately have links to articles. Then create or use existing modules to control placement and appearance of these menu/sub-menu, search, tags, etc. Templates are ultimate top level website files, which call and render these modules. You do not need to put any extra code in your template, as modules are rendered by joomla code itself, w/o any special code in your template.

website creation:

That is all we need to know for website creation. Remember that for building a simple website, we need to know very few (< 10) buttons of joomla. We'll just learn about the basic ones now, and then build on top of it later.

1. Log into joomla backend. On the very top, you will see 7 pull down menu:

  • system => 2 important links here are: "Control Panel" and "Global configuration".
    • CONTROL PANEL=> It's most important one, as it has links to articles, menus, modules and templates which is all we need to get a working website. In addition to these, it has links to "media", "categories", "user" and Global configuration settings. So, basically we do not even need to go to any of the next 6 pull down menu, as everything of interest is listed under "Control panel".
    • GLOBAL CONFIGURATION => This has global settings for various things on our website, and can also be accessed via "control panel" above. All default settings are good for us, no need to change anything here. Here we globally set options for everything as articles, modules, etc. This page also shows up when we click on settings on left top of page (only seen when we are editing something).
  • users => I just have 1 user for my website (which is administrator or me), so no need to mess with anything here. This is irrelevant for most single adminitrator websites.
  • menus => This is to manage menus and menu items. We use this to create any new menu or menu item. "Main menu" is already provided by default, so we can add all menu items to this "main menu". Add menu items as described above. We can add new menu also, so that we can other menu besides "main menu". This helps us to keep different menu items under different menu and show them differently by showing different menu on different pages.
  • content => We use this to add any new articles by clicking on articles. We can also upload files, images, etc by clicking on "media". Add articles as described above.
    • MEDIA: We can upload upto 10MB size of file in "media" section. You can change default of 10MB to any size by clicking on options on right, or going to system->configuration. The files can be pdf, jpeg, or any kind of video or text file. The path to this folder is "your-website-name/images". So, for a folder in "Media" called "sample" with file abc.pdf in it, the path would be www.mysite.com/images/sample/abc.pdf. We can also specify url as relative to main server path, so url as "images/sample/abc.pdf" is also recognized as a valid url.
    • UPLOAD FILES: The upload feature doesn't allow us to upload any html files, scripts/programs, or any other file that joomla thinks are unsafe. In order to upload such files, you will need to login to your website provider admin page, and directly upload those files in "images" folder. The dir for media is in your joomla_installation_dir/public_html/images.  So, for abc.com, it's in domains/abc.com/public_html/images. Goto this dir. You will see "index.html" file in every folder over here. You can create your folders here, though that's not preferred since then you will miss index.html file here. Best option is to create all your folders from joomla admin page using "create folder", and then upload files using this web server login page (and using file manager). Goto appropriate subdir in images, and upload files there. Or you can use ftp to transfer bunch of files in 1 go.
    • TEXT/IMAGE LINK: For putting link as a text for any article, we can choose Insert->Link drop down menu, and put the URL and the text and other details, and that produces a "text" link. To put a link as an image (instead of a text), first insert an image in an article. Once image is there, select image and click on "insert link". Now you can put the URL that you want your image to point to. If you toggle editor, you should see code like this: <a href=...> <img src=...> </a>. Or you can edit html code directly by going to html editor. This image serves as a thumbnail now. If you want to change shape/size of this thumbnail image, highlight the image, and click on edit image. Now you can edit this image just like any other image in your article. change Dimensions to 800 X 200 or something. Select "constrain proportions" if you want to keep original ratio of width X height of photo.
      • This image link is very helpful when you have a big image to display in your article, but you don't want that to take the whole page. Instead you want to show a scaled down copy of the image (i.e a thumbnail), and then when the user clicks on that thumbnail, it pops up the full size image. This looks neat and is very commonly used on websites. In such cases, the thumbnail image and the full image both have same url, and are pointing to same image on your joomla server.
    • LINK TO SECTION OF PAGE: One other kind of link you may want to put is a link to a particular section of a page. This can be done via putting "#" in href link and putting the "id" in the linked page. Again, use the html code editor directly to put the "id" in the section of page you are linking to. See in html section for details of syntax.
    • MODIFY HTML: Most of the changes you want to do in any article for which you don't have a button available is often easier to do by directly modifying the html code in the editor. Do not install any plugins, extensions or anything else for this.
  • components => we do not have anything of immediate need under this, so we can safely omit this whole section.
  • extensions => This has links for adding/updating extensions, modules and templates. Since we are going with template beez3
    • modules => This shows all modules, i.e main menu module, login module, search module. You can change the status of module to "unpublish" by clicking the status for that module. You can also add a new module by clicking on new module. We have to select a module type, and then fill in details. If we select module type as "menu", then on next page, we enter the menu that we want in this module, starting/ending levels, etc.
    • plugins => plugin is a kind of joomla extension. You can write your plugin to add new functionality. There are lots of useful plugins installed by default.
    • template => This allows us to see all available templates. If we haven't added new templates (by clicking on extensions->manage->install), we'll see 2 default templates provided by joomla. If we click on any template and click on "advanced", we can change template image, title, etc. Very important to change logo, title and description to match your website.
  • help => This has useful help links. Not needed for now.

2. Now, we add a new article using "content" pull down. After saving it, we add a new menu item using "menus" pull down. We add a new menu item under "main menu". We do not add a new menu yet, as we want to stick with default "main menu" provided. We just add a menu item under main menu. Once menu item is added, we now goto extensions pulldown and click on module. We see about 8-9 modules. We click on "main menu" module and make sure that selected menu on this is shown as "main menu". Next we click on extensions->template and make sure the right template is chosen.

3. Once article, menu, module and template are done in step 2 above, our website is ready to show our first article. Refresh your frond end page, and see if it shows your menu item and your article linked to that.

4. Under global configuration->site settings, we should change the name of site to our site. Remember to save any changes that you make, otherwise the change doesn't take effect. That's true for everything.

Helpful joomla settings:

1. Show menu at only 1 level of hierarchy at a time: One of the things that I've wanted to do is to hide sub menu on the main page, and only show the top level menu items. If the user clicks on the menu items, only then it should show lower level menu items. That way your menu items on main page don't show like 1000's of menu items at the same item, which will make it hard to read. One setting to achieve this is via:

 Extensions->Modules. Then select "main menu". It wil take you to "main menu" with 4 tabs, by default, module tab will be selected. On it, click on "show sub menu items" as "No" instead of default "yes". Do not change the start and end level, else sub menu items may not show at all beyond the end level, even on clicking on menu items on the main webpage (which is not what we want). "Click "save & close".

2. Increase timeout duration: set session timeout to 15000 as explained on top of this article. That will prevent a lot of your unsaved work from getting lost.

3. Reordering menu items: For menu items, you will notice that default ordering is decided by the order of creating the menu items. So, if you have menu item "physics", and you wanted to create sub menu items "chapter 1", "chapter 2" and so on, but if you created "chapter 2", before "chapter 1", then chapter 2 will show above chapter 1 in menu list. To fix this, we can change ordering of menu items at any time. This ordering is only available once you have already saved the menu item. After saving, you can click on "ordering" on right side, and click on menu item, after whch you want tyour current menu item to be placed. So, for ex, if you want to place chapter 2 and after chapter 1, then click on chapter 1, and save. Now, on the "ordering" pull down menu, you will see correct ordering. You can also check for correct ordering by refreshing your website page.

4 Separator between menu items: Sometimes, we want a separator between menu items, or something with no link, but just a bold text which separates one submenu from another. This allows for easy navigation. One way to do this is to create a new menu item, but instead of selecting that "menu item type" as "aricles", we select it as "system links". Here we can select it as "menu heading" or "separator" (both seem to do the same thing, but "separator" doesn't change text color/size as explained below. Maybe it has different class. So, stick with "menu heading"), and save it. After saving, we can put it in the correct order by selecting "ordering" on the right side and saving it again (as explained above).

The separator created above has same font size + color as other menu items, so it doesn't look like a real separator. To change the CSS properties like color, font-size, etc we need to edit the user.css file. Here's a link on how to do it: https://joomla51.com/tutorials/template-customization/applying-color-to-a-single-menu-item-in-joomla

In short, goto extensions->templates. goto "Beez3 details and files" or files of whatever is your current template. It takes you to the dir of that template. goto "css" folder, create a new file called "user.css", and write your custom css code here (seems like user.css is already provided in latest joomla version, so just add your code in there). This will overwrite other css proporties provided by default. We can make changes directly in any of the css files also, but then any changes you make to any of these files will be lost when you import new version of joomla (as beez3 and protostar templates are provided with joomla package and as such get overridden). user.css does not get over written by new installation, so any css changes to any file should be made in the user.css. user.css is the last css file to be loaded when joomla is running, so any settings in css/user.css will over write css settings in any other css file. So, don't change any other css file in template, just copy the code from other css file to user.css and make required changes in user.css.

In the ex provided in link, "item-210" is the class selector for menu items, where 210 is the id of that menu item as seen under that menu item listing.

If we want to apply a particular set of properties to all elements of a certain class, the better way would be to first find the class of such elements, and then apply the property. To find the class, we can look at html code of any page of our website. If I go to firefox browser, bring up my website and look up the source code, and search for any of menu item separator item (i.e finance), I see that the class for these is "nav-header", so I apply the properties to this class as below. In my case, I coded as below to change color to red and font-size to large, with padding to provide some extra separation b/w menu items and menu separator. 

.nav-header {
  color: red;
  font-size: large;
  padding: 15px 0px 10px 0px; /* provides top padding of 15px, while bottom padding of 10px. Top padding doesn't seem to work?? FIXME ??
}

5. Delete menu items and articles: When you trash a menu, menu item, article, etc, they don't get deleted, and it's possible to retrieve it from trash. However, sometimes when you try to create something with same name, joomla will not allow, even if the item is in trash. In such cases, you will need to delete that trash item permanently. The way to do it is to go to "search tools", when you are at at "all menu items", "all articles", etc, and then choose "status" as "trashed". then it will show "all trashed items". then click on "empty trash", and then item will be deleted permanently.

6. Featured articles: By default your first menu "Home" is shown on front page. Whatever is in the "home" article is what is displayed. However, you can make joomla show certain featured articles on this page, so that it looks more lively and recent, then having same stale content for ever. A very good link on how to do this:

https://docs.joomla.org/Managing_the_Featured_Items

In short, these are the steps. Change the menu item type of  "Home" menu item from "single article" to "featured article" by going to articles->featured articles. You can customize the payout of this page by changing settings under layout tab. You can also set the article order here. Now save "Home" menu item. At this point, you can go to articles page, and designate those articles as "Featured" which you want to display on "Home" page, and they will show up. You can also have a separate "Article" menu to show all articles under that menu. Then you can have a subset of these articles marked as "featured" and show them under "Home" menu.

One of the common things when displaying featured items is to show only a small introductory section of article, and not the whole article. That allows readers to see a lot of articles at once and they can click on articles of their choice. One way to achieve this is to add a "read more" by going to article and clicking "Read More" (3rd or last line on the article tab where you select size, link, etc for article). This adds a "red line" on your article content (in html code, you will see <hr id="system-readmore" /> at that place). However, when you go to your website, and look at this featured article, you won't see any text associated with this "Read More" tag. By default, "Read More" text is hidden. To fix this, goto menu for "Home" (or whichever menu is assigned for featured articles). There goto "options" tab, change "show read more" to "show" and "show title with read more" to hide (this may be needed only for beez3 template, since that template adds a title to be displayed by default on any read more button). You can change any other settings here too (like showing the article publish date, number of hits, etc). You can add a "introductory image" to an article, and that will show up alongside the article in featured article list.

One other way to show articles of certain type only is to put them into a ceratin category, and then create a menu with menu type as "articles->category list", and then under "choose a category", enter that category. then all articles belonging to that category will show up.

Writing new template:

Beez3 template:

We used the default "beez3" template above, but it's good to know how to write our own template. Before we write our own template, lets look at all the files on beez3 template. To do this, we goto extensions->templates->templates and click on details and files. It shows us few directories and bunch of files. We can also goto our webserver, and look in this folder to see these files:

website_server/public_html/templates/beez3/ => In your case the website server name may be something like abc.com

A short description of these files is provided on joomla website here:

https://docs.joomla.org/Understanding_Joomla!_templates

Important Folders:

css => lots of css files

html => lots of dir within it. contains template override files

images, javascript, languauge => these dir are not so important

 Important files:

templateDetails.xml => holds meta info related to the template

index.php => It's the main skeleton of webiste, and is the main file called for any page that is shown on the website. Fe details of this file:

  • Adds all the CSS stylesheets which are in css dir as layout.css, position.css, general.css. If user.css file is present, it's loaded at the end, so that it overrides all other css files
  • Main html file starts after that:

component.php, error.php => not so important for now

Few customizations to beez3 template:

1. Removing fontsize buttons: In beez3 template, by default, there is a "font size = Bigger Reset Smaller"  section on top right corner (just before the search box), which looks annoying to many. To remove this, we can just comment out this section in index.php. Just comment out line 130, which is "<div id="fontsize"></div> " as shown below (NOTE: it's html code in this line, so comment has to be in html syntax)

<!-- <div id="fontsize"></div> -->

2. Removing banner: In Beez3 template, you will notice a big banner with "joomla logo on left and and We are Volunteers message on right". The easiest way to remove this is to modify index.php file.

<div class="logoheader">  ..... </div> => This is the section that is responsible for putting that banner. Comment out the whole section as follows

<!-- <div class="logoheader">  ..... </div> -->

However modifying this file as above got rid of the container itself, which causes your top menu items (that can't fit the width of browser) to disappear when reducing the width of browser. The div container earlier used to hold those roll over of menu items from 1st line to 2nd line. so, instead we'll just remove the logo section and then take care of that div container other way.

Comment out the logo section as follows. (This preserves the div container as is. Don't comment out the div section, just the <h1> content inside it):

<!-- <h1 id="logo"> .... </span></h1> -->

 That gets rid of the logo, but the background picture is still there, as the css code for class="logoheader" is still present in personal.css. Here background image is set to "personal2.png" which is what we see on our webpage. This picture is not in media folder of joomla, but in "images" folder of beez3 template.

.logoheader {
    background: url(../images/personal/personal2.png) no-repeat right
     bottom #0C1A3E;
    color: #FFFFFF;
    min-height: 250px;
}

Instead of modifying the code here, we copy it in user.css, and modify it over there. That way it overrides whatever is in personal.css with the new settings in user.css. We should never modify any css file except for user.css (for reasons already mentioned)

In user.css, modify the background image to some other image, or just set background image to none.

.logoheader {
    background: url(../images/personal/bg2.png) repeat right bottom #0C1A3E; //here we set it to some other image already present in that folder with repeat option so that the smaller sized image repeats itself. NOTE: bottom #0C1A3E is part of "background" and should have been on same line for clarity (see ;  in end)   
    color: #FFFFFF;
    min-height: 250px;
}

If we don't want to put a picture at all, and just put some color or leave blank, we can set that as follows:

.logoheader {

    background-image: linear-gradient(red, yellow, green); //Use this option if you want to have some colorful design there. Many more functions available in CSS documentation
    background-image: none; //Use this option if you want no image at all
    color: #FFFFFF;
    min-height: 50px; //Change the height to 50px, since we don't want a big blank section in there. If we set it to 20px or less than only 1 row of "menu items" can fix, which causes menu items to dis appear if width of browser reduced too much. 50px allows you to have 2 rows, so it's somewhat better as it allows more menu items to show until they disappear
}

3. changing width of page: By default, you will see that your joomla website doesn't extend all the way to the right side, i.e if you keep on increasing the width of your browser, the website stops growing in width on right/left side, and will instead start showing blank on right/left most side. We'll need to change width attribute, and the place to change it is in user.css of the beez3 template (as explained under item 4 above in joomla settings). A sample code to change width is as below:

/* copied from file css/position.css, changing max-width from 1050px so that 15 inch laptop screen can be full width */
#all {
    margin: 0 auto;
    max-width: 1550px;
    padding: 0;
    text-align: left;
    font-size: 0.8em
}

/* copied from file position.css, above code didn't change article width, just page width. This changes article max width from 660px */
#wrapper2 .item-page {
    max-width: 1260px
}

/* copied from file position.css, changed width from 21% and added max-width to keep navigation menu on left from getting too wide */
.left1 {
    padding: 0;
    float: left;
    margin: 10px 3% 10px 0;
    width: 16%;
    max-width: 80px
    position: relative
}

4. Adding footer: Beez3 template doesn't show footer on webpage by default. However it's really easy to add footer via 2 ways: First is where we use the footer module that is already provided. We click on "Modules", then new which shows all modules available. We choose "footer" module and place it in position 14, which will place the footer on bottom of page. Second way of adding footer by choosing "Custom" when clicking on new module. This custom footer module can be written by putting in whatever content you want in the editor of footer module. If you want the footer content to be centered, then you can put this html code in Editor:

<p style="text-align: center;">  Copyright © 2020 Maaldaar </p>

One thing to note is that since we extended the width of our website, we should increase the width of footer module too by adding this in user.css:

#footer {
  max-width: 1550px
}

A very good video for this is here: https://www.youtube.com/watch?v=orWoxsjD170

5. Changing default joomla icon to your own website icon: Beez3 template (or any template) by default shows joomla icon on the tab where you open website. It's called "favicon" (favorite icon). You would want to change it to some other custom logo or picture. You can create or find an image for your logo, and then goto https://favicon.io/, and then use the converter over there to create a ico file from png or jpg file. Once you have created ico file and downloaded it in your local computer, you now goto your webserver hosting panel to upload this file. You can't use joomla admin interface of your website to upload this file, as it doesn't support uploading these kind of files. Once you goto dir: templates->beez3 of your server website dir, you need to upload this file with the name favicon.ico in this dir. any other name won't work. Now if you refresh your website multiple times, you should see your new logo on tab of your browser.

More details here: https://docs.joomla.org/Special:MyLanguage/Changing_the_site_favicon

6. changing page title to be displayed on browser for any menu item: By default, page title shows the menu name. Sometimes we may want to change this default behaviour. For ex: menu item "home" (which is the homepage for yur page) shows "home" by default on the title of the page. We may want to change it to the name of the website. For that, go to menu item "home", then click on tab for "page display" and change the title to "maaldaar" (or in your case your website name). Now whenever you goto your website, the default title shown on browser tab will be "your website name". NOTE: "home" title still remains for the menu item.

7. 404 Error page: By default, 404 error page shows joomla error page, with all joomla content. To customize it, you need to create a custom 404 page, and then link to it in error.php file. There is a good article about this: https://www.joomlashack.com/blog/joomla/404-pages/

First create an article with any title for your 404 page, then create a menu item for that article. "Unpublish" it so that it doesn't show up in "menu list". Now goto "error.php" file in Extensions->Templates->Templates->Beez3_details_and_files. Over there enter the line as below just after "defined('_JEXEC') or die;"

 if ($this->error->getCode() == '404') { header('Location: /theURLtoYour404Page'); exit; }

Leave rest of the code unchanged. You can remove it too, but it doesn't matter, as we already did a exit in the line above. So, rest of the code never gets executed. To remove the top bar from our error page (" the box that shows "Published, Hits, last updated, etc), we have to hide all those from showing by changing settings for this article on artcle page or menu settings.

New template from scratch:

The best way to learn templates is by writing your own template. This link explains it very nicely:

https://docs.joomla.org/Creating_a_basic_Joomla!_template

Once you have written your template, set your default template to your own template, and test it on your own website. Even though your website won't look fancy, but it will work !! I haven't gotten time to write my own template yet, but someday I will.

 

PrimeTime: gate level STA tool. dynamic analysis requires input vectors and simulating those, so may not capture all possible paths. PT has most of the same timing cmd as used in DC.

  •  It takes gate level netlist in: .db, .v, .vhdl format
  •  It takes delay info in std delay format (SDF). Both net and cell delay are in this file. This file is optional. In absence of this file, SPEF and .lib files used.
  •  It takes parasitic data in std parasitic exchange format (SPEF). Only net delays can be calculated here by using parasitics from SPEF. when SDF not vailable, SDF generated from SPEF, and used for net delays. It takes cell delays from .lib.
  •  It takes timing constraints in synopsys design constraint (SDC) format

For prelayout (synthesized netlist with no PnR) timing analysis, cell delay come from .lib and net delay come from wire load models which estimate delays based on estimated cap and res of nets. PT supports use of timing models to represent chip submodules or std cells. Liberty model (.lib) is widely used model that it supports.

PrimeTime Invocation: Explained under STA PT flow on how to bring up PT tool.

Primetime cmds:

Looking at PT manual for 2018 (it can be downloaded from synopsys solv, it shows around 700 cmds, but only 100 or so cmds are useful. Some of the important cmds from Synopsys DC (Design Compiler) are used in PT also. A subset of cmds from DC have been used to form sdc standard (see in sdc section) for design constraints.

Cmds in synopsys can take any input which are collections or list of object names, and returns o/p as collection or list of object names. Thus there can be 4 categories of cmds as shown below. Which cmd falls into which category depends on the particular cmd.

  1. i/p=collection, o/p=list
  2. i/p=collection, o/p=collection
  3. i/p=list, o/p=collection
  4. i/p=list, o/p=list.

NOTE: many cmds which take collection as an i/p can also take lists as an i/p.

1. report_* cmds (100's of them): These report cmds are used for reporting anything about the design on the screen. They take as i/p list of objects, and return o/p as list.These cmds should be used for reporting anything, as they dump names on the screen, or can be redirected to other file. Since, we get these o/p as list (not collection), so they can be viewed directly w/o any further processing. report_timing is most useful cmd here. You should first try to use "report_*" cmds, and if they don't give you the info you need, then you should use "get_*" cmds (explained below) as they are more comprehensive (also get_* cmds are more cumbersome to use)

2. get_* cmds (50 of them): These take as i/p collection or list of objects, and return o/p as collection. Since the o/p is a collection, we have to post process the collection by taking it thru a for loop and then print whatever we want to print. These are useful when used within a script, but not very useful for quick reporting purpose. Do not use this when you just want to display something for reporting. Use report_* cmds for reporting purposes. These get_* are most commonly used cmds for writing tcl scripts within PT.

3. remove_* cmds (100's of them): These are used to remove something from design all together. These are not used that often. They are mostly used used during debug, when we have added some object incorrectly, and we want to remove it. Or we want to start from clean slate, and so use remove_* cmd in the very beginning to remove things in design that we don't want.

4. design related cmds: create_*, read_*, write_*, set_*, reset_*. These cmds are most useful for setting constraints on design, so that we can analyze timing paths in design.

  • set_* cmds (100's of them) => Thsese are used to set various attr for design.
  • reset_* cmds (20 or so cmds) => Just like remove_* cmds, they are used during debug to reset something previously set using the set_* cmd. So, reset_* cmds just undo the effect of corresponding set_* cmd.
  • read_* cmds (20 or so cmds) => Used for reading external files, design, etc.
  • write_* cmds (20 or so cmds) => Used for writing to o/p files. These are counterpart of read_* cmds.
  • update_* cmds (update_timing, update_power, update_noise) => These cmds are used to update timing, noise or power for the current design. Design is automatically updated if needed for reporting that aspect of design (without us explicitly needing to do it). So, we'll use these rarely.
  • current_design / current_instance => shows current design that it's set to. It may be top design, or one of the hier. It's
  • current_instance =>

5. all_* cmds: Couple of cmds here to see all ports, regs, clks, etc. Helpful and commonly used.

6. list_* cmds:

7. check_* cmds: These are used to check some important aspect of design. These cmds are a sanity check to make sure there's nothing wrong with that aspect of design. We should always look for errors/warnings in these check_* cmds, as they check_timing is most imp cmd here. There are just 6-7 cmds here as check_power, check_noise, check_constraints, etc.

8. misc cmds:

A. alias/unalias: these can be used to set alias just as in Linux, so that we don't have to type lengthy cmds. We can't set alais for an existing cmd. We have to unalias that cmd, and then alias it again.

ex: alias rt { report_timing -max_paths 100} => Now whenever we type rt as the first word in pt_shell, it will automatically expand it to the cmd shown.

B. ETM: Extracted Timing model. As explained in liberty section, we can generate an ETM for any design. The PT cmd for this is:

extract_model => this generates an ETM for current design from it's gate level netlist. The generated model is in .db format (synopsys proprietary liberty format), and exhibits the same timing characteristics as the original design and therefore is used instead of the gate-level netlist for purposes of static timing analysis. Look in PT manual for various options. There are also 30 or so attributes or variables that can be set to control what kind of timing data is present in ETM .db file. These are shown in PT variable attribute manual. These attr are extract_model_*

PT CMD PRECEDENCE: In PT you may write same cmd multiple times on different objects, and all of them are applied simultaneously. What happens if the same object has 2 conflicting cmds applied to it? Which on takes precedence? We don't do it based on "whichever last one overwrites earlier ones. Reason for not doing this is this. Let's say you have a "set_max_delay" set on a set of paths, and then later on you do "set_max_delay" on some overlapping paths from before. PT has no way to overwrite previous cmd, since cmds may be undone by doing "remove_*" cmds only. So, PT has to look at all cmds simultaneously, and then come up with a strategy of prioritizing cmds. This  PT manual says that cmds which are more specific will take precedence over cmds that are more general. This holds in general for all cmds with the exception of "set_false_path". See in set_false_path section for that.

Ex: group_paths cmd sets groups. We can have paths overlapping in 2 groups. i.e

1. group_path -name io_to_flop -from [all_inputs]
2. group_path -name io_to_io   -from [all_inputs] -to [all_outputs]

Here paths from input are part of both groups. However, group2 (io_to_io) is more specific. So, all paths from input which goto output are put in group2. Then remaining paths from input are put in group1 (io_to_flop).

However, if 2 constraints are conflicting and none of them is more specific than the other, then I'm not sure how the precedence rules work ??

 

 

USA Income and Taxes Data:

To look at the US economy, we have to first look at the total income of all the people living here, as well as their wealth.

US income/wealth percentile:

This data is for around 2020-2025. 

There's a neat income percentile on this link: https://dqydj.com/income-percentile-calculator/

There's also a wealth percentile here: https://dqydj.com/millionaires-in-america/

We saw in other USA sections, that there are about 130M households and about 150M working population in USA. Income percentile refers to percentile in these 150M workers, i.e top 5% means in the top 7.5M earners. Wealth percentile refers to these 130M households, i.e top 10% means you are in the top 13M households. Wealthier households are usually middle age or old age household as it takes 20 years or more to accumulate enough wealth to get to top percentile with a day job.

  • 0%-70%: We see that every $1K in income gets you 1 percentile higher upto $70K in income. That means if you have $1K in income, you are in bottom 1%, while if your income is $70K, you are in 70 percentile. So, 0% to 70% goes up linearly with income. After this starts slight non linear curve.
  • 71%-80%: Here, income goes up by $2K for every 1%. 70% starts at $70K in income, and 80% ends at $92K in income.
  • 81%-90%: Here, income goes up by $4K for every 1%. 80% starts at $92K in income, and 90% ends at $132K in income.
  • 90%-95%: Here income start going exponentially higher, and you need to make $10K more to get up by 1%. Income start at $132K and end at $186K. About 15M or 12% households are millionaires, meaning most of them are in this income group starting at 90% and above.
  • 96%-99%. Here income go even more exponentially higher, and to make it to the top end of this income group, you need to make at least $500K. About 5M or 4% households have $4M or more in assets, and they are usually earners in this group.
  • Top 1%: These are the richest people, who not only make > $500K/yr, but also have wealth > $10M. About 1M households have wealth > $10M, with 100K households having wealth > $50M, and 30K households having assets > $100M.  These 30K huseholds are who's who in America, and are generally well known. This is the chart showing top 1% of households income wise per state. As can be seen, in richer states as Connecticut, Masschusetts, California, New york, your household needs to being in close to $1M, while in poorer states as West Virginia, Mississippi, New Mexico, etc, just $0.5M will get you there.

Bottom 90% of the people are so poor they they live and die as slaves. More on this in another article. "For the bottom 60% of U.S. households, a "minimal quality of life" is out of reach, according to the group, a research organization focused on improving lower earners' economic well-being. " Link => https://www.cbsnews.com/news/cost-of-living-income-quality-of-life

 

 


 

IRS income data

A lot of tax and income data available on IRS website. We want to just get an idea of how income of the population lines up.

IRS data taken for years, 2017, 2011 and 2007 (taken from here: https://www.irs.gov/statistics):

Types of returns filed: 3 major kind of tax returns filed, which account for 80% of tax revenue:

A. Individual tax returns: Filed by individuals like you and me. These are the 1040 forms filed. I'm showing data for 3 years: 2007, 2011 and 2017.

2017:

Selected data pulled from IRS stats file here: https://www.irs.gov/pub/irs-soi/17in14ar.xls

This is the 1040 form which the above table refers to: https://www.irs.gov/pub/irs-prior/f1040--2017.pdf

152M individual tax returns (form 1040)  filed.  73M were returns of single persons, 55M were returns of married filing jointly and 22M were returns of heads of households, and 3M were married filing separately.

AGI (line 37 in form 1040): AGI is Adjusted gross income and is line 37 in form 1040. Total AGI was $11.1T.

Number of returns with AGI > $500K was 1.5M or about 1% of total returns. 1M of these returns had AGI < $1M. So, once you start hitting AGI of $1M, you get in top 0.3% of population. Total cumulative AGI for this group is $2.3T (avg AGI = $1.5M). Only $0.9T of this income came from wages/salaries. $0.6T came from sale of capital assets, $0.1T from dividends, $0.6T from partnerships and S corps. In fact, as you go higher up in AGI, higher percentage of income comes from passive assets and not from wages (for people with AGI > $10M, income from assets is 6X their wages. So, decline in price of stocks/equities will hurt this top 1% very disproportionately as 25% of their AGI will evaporate in absence of capital gain (as they can't sell stocks for profit when markets are down for the year).

Number of returns with $200K < AGI < $500K were 6M or 4% of returns. So, an AGI of $200K or more places you in top 5% of working US population. Total cumulative AGI for this group is $1.8T (avg AGI = $300K).

Number of returns with $100K < AGI < $200K were 20M or 14% of returns. So, an AGI of $100K or more places you in top 20% of working US population. Total AGI for this group is $2.7T (avg AGI = $130K).

Number of returns with AGI < $100K: Remaining 125M returns fell in this category. This represents bottom 80% of the population. These are the poorest people with very little savings, and little assets. Their combined AGI is $4.5T (avg AGI = $36K), which is about half of tthe total AGI of entire US workforce. Their AGI is mostly comprised of wages, and SS/pensions. 80% of AGI, or about $3.5T comes from wages. Remaining $0.5T comes from retirement accounts, and $0.5T from Social security benefits. So, most of the retired people fall in this category. Also, most of the youngest workers under 26 years of age fall here too, as they are just starting. 2M returns are for people under 18 years of age, totalling $0.01T while 23M returns are for people between age of 18 and 26 years of age, totaling $0.45T. So, 25M returns are from people under 26 years of age, who mostly fall in AGI < $100K. When you hear about unemployment in news, this is the group that is most vulnerable, and comprises of most unemployed people. This bottom 80% drives the so called economy, as whatever they make gets spent, with negligible savings. About 50M people have non taxable returns, meaning they owe $0 in federal income taxes. Looking at table about 54M returns had AGI < $25K (or income below poverty line for a family of 4, with 2 kids and 2 adults. For an individual, poverty threshold is at $13K), so most likely these are the people having non taxable returns, since std deduction and many other credits can wipe out taxes. US census.gov website shows 11% of people below poverty line, while here we see 35% of population with AGI below $25K. Reason is many individuals filing return as single person are above poverty line, since poverty threshold for households with single person is at $13K.

Social security website shows that about 6% of returns filed have earnings over maximum taxable limit for social security for each year since 1980. Congress adjusts the Social Security limit every year to keep this 6% number constant. For 2017, the limit was $127K. This limit has doubled from $68K in 1998 to $138K in 2020, implying that wages/salaries for top 6% are rising at an annual rate of 3.5%.

https://www.ssa.gov/policy/docs/population-profiles/tax-max-earners.html

Bottomline: So, to be in top 5% and to stay there, you need wages > $130K and AGI > $200K for year 2017, and both of these need to go up by about 4% every year starting from year 2017. Also, you need to have extra income from dividends, interest, stocks sale, comprising at least 1/2 of what you get in wages/salaries. So, that way your AGI comes out to 150% of your wages.

AGI componenets: Various components of AGI are as follows:

wages/salaries (line 7 in form 1040) = $7.6T (126M returns), So, that leaves 26M returns (since total returns=152M) with no wages/salaries, which is mostly older retired people and disabled people. 29M returns had Social security benefits amounting to $0.6T. 15M returns had IRA distributions totaling $0.3T. These are mostly older people not in workforce. This implies that 26M returns are filed by people not employed (i.e old people living off social security, disabled people, etc). Looking at other table, we see that 26M returns were for people 65 and older, with collective AGI of $2.1T, out of which only 9M had salaries and wages totaling $0.5T.

interest (line 8 in form 1040) = 0.16T : taxable interest = $0.1T (44M returns), tax exempt interest = $0.06 (6M returns), => about 35% of returns have money earned thru bank account CD, savings a/c, etc. So, 2/3 of workforce don't even have bank accounts or don't earn any interest. Of the bottom 80% with AGI < $100K, 26M had taxable interest income totaling $0.025T. So, 80% of the interest income is attributed to top 20% of people, since they have savings. However, total deposits in US banks is over $10T, so $0.1T interest implies 1% interest rate, which is the highest rate banks offered in the last decade. So, basically income from interest has evaporated.

dividends (line 9 in form 1040) = 0.28T : ordinary dividends = $0.28T (28M returns), qualified dividends = 0.22T (26M returns), => about 20% of all returns have money earned thru stocks, since qualified dividends mostly comes thru stocks. That's a very high percentage of people invested in stock market thru their personal ccounts, considering that only 35% have bank accounts. Of the bottom 80% with AGI < $100K, 16M had ordinary dividend income totaling $0.05T. Here, just as in interest income above, 80% of the dividend income is attributed to top 20% of people. People with AGI in range $100K < AGI < $500K made about $0.1T in dividend. Assuming most of the dividend comes from US equities, it implies, that top 20% of the people have a lot of money invested in equities/stocks. Looks like all of America owns equities, with top 1% owning 50% of equities. Also, let's assume that $500B paid in dividends was paid by US companies in 2017 (no reliable source found for this but that's the range dividends have been around 2015-2020 ? FIXME?). Since ordinary dividends are $0.28T, then this implies that about 50% of US equities are owned by people in their personal accounts, while remaining 50% is held in their retirement accounts. Note that most of the equities held in personal accounts is for people in top 10%.

sale of capital assets (line 13 in form 1040, schedule D) = $0.8T (17M returns), Of these, about 6M returns filed for net cummulative loss of $13B. Reason for such low net loss is because losses are limited to $3K/return, so net loss could not exceed 6M*$3K = $18B in any given year. Again, people with AGI <$100K (or bottom 80%) of people had just $0.05T in cummulative capital gain, which is just 5% of all capital gains. Top 1% had capital gains totaling $0.6T or 75% of all capital gain. Sale of capital assets include house, stocks, and many other assets. Looking at tables on irs website for "capital gain/loss by asset type for year 2012" didn't give a clear picture on how much of this is from stocks and how much from other assets.

partnership and S corp income (line 17 in form 1040) = $0.7T (7M returns). Many individuals also form partnerships to open corporations. The profit from these corporations pass thru their individual income tax return,

rental and royalty income (line 17 in form 1040) = $0.05T (11M returns)

Of 26M returns that didn't have wages (152M-126M), most of them are either old/disabled people collecting SS/IRA/401K, or people with their own business

business income (line 12 in form 1040) = $0.35T (26M returns). Surprising that 20% of workforce have their pwn business. Probably a lot of mom/pop shops or franchises. What's interesting is that about 6.5M or 25% of these business returns had net loss. If that's true then a lot of these individual business would go bankrupt every year.

pension/annuities (line 16 in form 1040) = $1.2T (30M returns). distributions from 401K plan are reported under this on line 16a, 16b of form 1040. 18M returns were from people over age of 65, totaling $0.7T, while 6M returns were from people between age of 55 to 65, totaling $0.3T. As expected, 80% of this income is attributable to people in retirement.

taxable IRA distributions (line 15 in form 1040) =$0.26T (12M returns), here distributions from personal IRA accounts (not thru work) are reported on line 15a, 15b of form 1040. 11M returns were from people over age of 65, totaling $0.21T, while 2M returns were from people between age of 55 to 65, totaling $0.05T. As expected, 80% of this income is attributable to people in retirement.

Social security benefits (line 20 in form 1040) = $0.6T (29M returns),

unemployment compensation (line 19 in form 1040) = 0.1T (13M returns) and others were misc items.

Of this AGI, 100M returns applied for std deduction of $0.75T, while 45M returns applied for itemized deduction of $1.2T. So, Taxable income was $5.7T after accounting for deductions, exemptions, tax credits, etc. IRS collected a tax of $1.05T (~20% of taxable income).

2011: 145M individual tax returns (form 1040)  filed.  67M were returns of single persons, 53M were returns of married filing jointly and 22M were returns of heads of households, and 3M were married filing separately.

Total AGI was $8.4T. Various components of this are:

wages = $6.0T (120M returns),

interest = 0.2T : taxable interest = $0.12T (52M returns), tax exempt interest = $0.07T (6M returns), => about 35% of returns have money earned thru bank account CD, savings a/c, etc.

dividends = 0.35T : ordinary dividends = $0.2T (28M returns), qualified dividends = 0.15T (25M returns), => about 20% of returns have money earned thru stocks

capital gains = $0.4T (22M returns),

partnership and S corp income = $0.4T (8M returns),

rental and royalty income = $0.05T (11M returns)

Of 25M returns that didn't have wages (145M-120M), most of them are either old/disabled people collecting SS/IRA, or people with their own business

business income = $0.3T (23M returns),

pension/annuities= $0.6T (27M returns),

Social security benefits = $0.2T (17M returns),

taxable IRA distributions=$0.2T (13M returns)

unemployment compensation = 0.1T (13M returns) and others were misc items.

Of this AGI, 100M returns applied for std deduction of $0.75T, while 45M returns applied for itemized deduction of $1.2T. So, Taxable income was $5.7T after accounting for deductions, exemptions, tax credits, etc. IRS collected a tax of $1.05T (~20% of taxable income).

2007: 142M individual tax returns (form 1040)  filed.  Total AGI was $8.7T. Of this wages = $5.8T, capital gains = $0.9T, pension/annuities= $0.5T, partnership and S corp income = $0.4T,  business income = $o.3T, taxable interest = $0.25T, ordinary dividends = $0.25T, Social security benefits = $0.2T, taxable IRA distributions=$0.2T and others were misc items. Taxable income was $6T after accounting for deductions, tax credits, etc. IRS collected a tax of $1.1T (~20% of taxable income).

B. Employment tax return: This tax return has to be filed by employers to pay social security and medicare tax to the government. Note that each employed person in USA is supposed to pay 12.4% SS tax and 2.9% Medicare tax. If you are self employed, you pay all of this, but if you are employed by someone, you pay one half and your employer pays the other half. Most of the people are employed by someone else, so their employer files tax form 940, and pays this tax to IRS. We as employees never see this tax return being filed, as money is already taken out of our paycheck by the employer. Employer then files this separate tax return paying his share (6.2%+1.45%)as well as paying the money that he took out of our paycheck (6.2%+1.45%). Self employed people file this return themselves. There were 30M such tax returns filed for 2007. Employers don't file separate tax return for each employee, but combine it in one for all their employees. So, this tells us that there were 15M employers (as 15M are self employed people. If we assume that each employed person paid about $6K in SS+Medicare tax (assuming median income of $50K), then for 150M employees (for 2008, workforce was about 155M), that would be about $1T in employment taxes.

C. Corporations: 7M corporate tax returns were filed for 2008.

D. Partnership: 3M partnership returns were filed for 2008.

There are 2 kinds of business: unincorporated and incorporated.

A. Unincorporated business: The revenue, expenses and income of business flows thru personal tax return. It has unlimited liability, is unable to defer taxes, and must file income taxes based on calender year. It's simple to set up these kind of business, and they have no payroll. These business are also called non employer or self-employed firms, and are genrally excluded from all business statistics from census bureau, as they account for only 3% of all business receipts in USA.

UnInc business, are further subdivided into 3 types as per IRS classifications:

1. Sole propertierships: someone who owns an unincorporated business by himself or herself. Generally non-farm sole propertiership are included in all stats.

2. Partership: relationship existing between two or more persons who join to carry on a trade or business. Each person contributes money, property, labor or skill, and expects to share in the profits and losses of the business. Partners may be individuals, corporations, other partnerships, tax-exempt organizations, nominees, or other legal entities. Partners are not employees and should not be issued a Form W-2. The partnership must furnish copies of Schedule K-1 (Form 1065) to the partners.

3. S-corporations: These are an alternative to Inc corp, but they should have no more than 100 shareholders, and have one class of stock. S corporations avoid double taxation on the corporate income, as income is passed thru tax on individual's income tax return.

B. Incorporated business: Usually represented by Inc at the name of business. The revenue, expenses and income of business flows thru corporate tax return, at corporate tax rates (usually 35%). The profit is again taxed when it's distributed to shareholders as dividends. It has unlimited liability, credit proofs its shareholders, is able to defer taxes, and file income taxes based on their own fiscal year which may or may not coincide with the calender year. It's complex and expensive to setup these kind of business, and it has payroll. There is only 1 type of Inc business, known as C corp.

LLC: A Limited Liability Company (LLC) is a business structure allowed by state statute. LLCs are popular because, similar to a corporation, owners have limited personal liability for the debts and actions of the LLC. Other features of LLCs are more like a partnership, providing management flexibility and the benefit of pass-through taxation. The federal government does not recognize an LLC as a classification for federal tax purposes. An LLC business entity must file a corporation (C-corp or S-corp), partnership or sole proprietorship tax return.

Note that sole proprietorship, partners, S-corp are included above in individual income tax returns as the income from these flow through individual tax returns of owners and not through corporate tax on business (as in C-corp) itself.

Workforce

Workforce is comprised of people hired by government or by private companies. Total population of USA is 315M, of which 50% don't go for work. Of this 160M who don't go for work, 45M are elderly (65+), 75M are kids (<18) and 50M are stay at home mom/dads and disabled people. Remaining 155M want to work and are either working or looking for work. They are called the civilian workforce (Total number of people in the non-civilian workforce is 1.5M who are hired by the military). Since unemployment or underemployment is about 10%, we can assume that only 140M people are employed. Lets break this number further down:

I. Federal government: employs about 2M civilian workforce (on top of 1.5M military workforce), and is the largest employer in USA. It excludes postal service employees of usps. avg salary is about $100K.

II. State and local government: employs about 17M civilian workforce. 7M are employed in schools as teachers and support staff, 2.5M in protective services (including police officers, fire fighters, and correctional officers), 2.0M in higher education, 1.4M in health care (including nurses and other workers at public hospitals and clinics), 1M in libraries, housing, others, 0.8M in transportation (including road maintenance workers and bus drivers) and remaining 2M in all other professions. Link is:   http://www.cbpp.org/cms/index.cfm?fa=view&id=3410. about 35% of state and local government spending is on wages of these employees, excluding healthcare and retirement benefits. Including these benefits, about 45% of the spending is on wages.

so, about 15% of the civilian workforce is employed by Government (public employees), while remaining 85% is employed by private companies or are self employed. About 15M (10% of civilian workforce) is self employed, with 10M unincorporated self-employed (often work by themselves without employees) and 5M incorporated self-employed (often have employees).

In year 2007, there were total of 27M private non-farm business in USA, out of which 21M were unincorporated businesses (no employees), and 6M incorporated business (with employees). UnInc business brought in $1T in business sale receipts, while Inc business brought in $30T in sales receipts. This is the link from census bureau: http://www.census.gov/econ/smallbus.html

These 21M business which had no employees include the self-employed, freelancers, independent contractors, sole proprietorships, family-owned businesses, LLCs, corporations, S-corporations or partnerships. So, the number of unincorporated self employed above implies that out of 21M businesses with no employees, only 10M are actually a day to day business with an owner. The remaining 11M are just corporations, partnerships built for tax/liability advantages.

Ap per IRS data from 2007 (http://www.irs.gov/pub/irs-soi/07ot3naics.xls), there were 32M business returns filed (instead of 27M that's expected). These had business receipts of $30T, with net income of $3T. Of these, 23M returns were for no-farm sole proprietorship, which had $1T in business receipts and 0.3T in net income. 3M returns were filed for partnerships (Form 1065). The number of partners on these returns was 18M. Most of these partnerships are in Finance, Insurance, Real estate, construction, retail and other services. Net revenue from these partnerships was $5.9T (business receipts = $3.9T, portfolio income from interest/dividend/capital_gains=$1T, others = $1T) while net income was $0.7T ($0.3T was ordinary income while $0.4T was from dividends, interest, royalties, rental income, etc). total assets of these partnerships was $20T. About 6M corporations in USA (includes both C-corp and S-corp), but 4M of these are are S-corp or other pass through entities (like REITs, RICs, etc) which pay no corporate tax. These S-corp had business receipts of $6T, with net income of $0.4T. About 2M were C-corp, for which corporate income taxes were filed, which paid tax of $0.3T on taxable income of $1T (real income was $1.5T, but taxable income came down with deductions/credits. corporate tax rate is 35%). Total assets was $82T. Total revenue was $22T, of which business receipts were $18T. Total revenue is more than GDP as corporations sell not only to consumers, but also to each other.

So, if we take out these 15M self employed from the civilian workforce, we are left with 120M people, or about 75% of the workforce. This 75% of the workforce is hired equally by small business (Business having less than 500 employees) and large business (Business having more than 500 employees). Of the 6M businesses with employees, only 20K businesses had more than 500 employees.   These small businesses employed about 60M people, while large businesses (about 20K in number) employed the remaining 60M ppl. So, these small businesses had about 10 employees on avg, while large businesses had about 3K employees on avg. Large business are usually all public or large private companies, which have chains throughout the country. Small business are the ones run by a owner, hire few employees, and typically include all mom-pop stores that you see around, gas-stations, restaurants, motels, personal care, house maintenance contractors, or small internet companies.

Among the large businesses, Walmart is the largest private employer at 1.8M employees, followed by Mcdonalds and UPS each at 0.5M employees. USPS employes about 0.6M people and is the second largest cvilian employer behind walmart, but is generally not counted as a private employer. It has the backing of federal government. The top 50 largest employers, excluding usps, employed about 12M people. link: http://nyjobsource.com/largestemployers.html . Most of these companies are retail companies, and as such the salaries of majority of their employees are < $50K/year.

Employment taxes (FICA and medicare tax from both employees and employers): Total tax of $0.9T was paid to IRS.

Total money earned by individuals, corporations and Government:

individuals income=$7.6T(money kept in pocket) + $1.1T (federal tax) + $1.4T (approx state/local tax incl property taxes, as state/local tax for 2012 was $1.4T) + $0.9T( assumed contributions of 10% to IRAs, health insurance premiums,etc which are excluded from AGI)=$11T.

Corporations income=$1.1T(money kept in corp reserves) + $0.3T (federal tax) + $0.1T(dividends paid).

Federal Government income: total taxes collected by IRS = $2.5T amounting to 18% of GDP.  individual income tax=$1.1T, Corporate income tax=$0.3T, Employment tax=$0.9T, Excise/gift/estate tax=$0.1T, others=$0.1T.

sum total of all money = $11T(individual income) + $1.5T(corporate income)+ $1T(federal government income from Employment tax + other tax, which weren't included in individual or corporate income) = $13.5T (while GDP was $14T)

individual expense:

corporate expense:

Federal Government Expenses: total = $3T. defense = $0.7T, healthcare = $0.7T($0.4T to medicare for 45M elderly/disabled and $0.3T to medicaid/CHIP for 60M low income people),   pensions=$0.7T(mostly Social security avg check of $1.2K/month to 35M retired workers and 10M spouses/children of retired/dead workers, and 10M disabled workers), welfare=$0.4T (unemployment assistance, food stamps, low income assistance,etc) interest=$0.2T(on debt of $9T) , education=$0.1T, transportation=$0.1T and others=$0.1T.

GDP:

USA nominal GDP was approx $?T in 2011 ($14T in year 2007).

GDP components: GDP can be measured using income approach (sum total of income of all individuals living in a country during 1 year) or using production approach (Market value of all final goods and services calculated during 1 year) or using expenditure approach (All expenditure incurred by individual during 1 year ). Mostly expenditure approach is used, components of which are shown below:

A. personal consumption: = 70% of GDP. money spent by individuals on various items and services.

1. services: 40% of GDP. The two largest components are real estate (10%) and health care (12%)

2. non-durable goods: 20% of GDP. The three largest components are food (10%), clothing (2.7%) and fuel (2.4%)

3. durable goods: 8% of GDP. autos (3.6%) and furniture (3%).

B. Private/Business investment: 16% of GDP

1. Non-residential: 12%

2. Residential: 4%

C. Government investment: 19% of GDP

1. Federal Govt: 7% of GDP. defense spending=5%, non-defense spending=2%

2. state and local Govt: 12% of GDP

D. Import/export: -5% of GDP.

1. exports: 12% of GDP, with goods being 9% of GDP and services comprising 3%.

2. imports: 17% of GDP

 


 

 

Population:

Population in USA is about 310 Million as per 2010 census. ( https://www.census.gov/data/tables/time-series/demo/families/households.html ) There are 113 Million households (house, apartment, or any separate living quarter in which any number of people can live). 230M people are white (of these 50M people are of hispanic origin), 42M are black/african, 15M Asian (Chinese=3.5M, Indians=2.9M, Filipino=2.6M, Vietnamese=1.7M, Korean=1.5M) and 3M American Indians. Note that these refer to origin or race, not citizenship. So, a baby born to chinese citizens here will be a considered of chinese race, even though he and his parents are citizens here. About 60M are households with married couples, while 20M are other family households. About 40M are non-family households (people from different families sharing an apartment, or a person living by himself in a dorm, apt etc but who doesn't consider himself a family. Ex: kids in dorms, unmarried people with new jobs sharing apt, etc). Number of households increase by nearly 1M per year.

Population wise, California, Texas and Florida have been top 3 states (all with relatively warmer weather than rest of the country, so they have also been top states for retirement). 25% of US population lives in these 3 states.

Population growth:

world population is growing by 1.2% yearly as of 2010 (2.2% is birth rate, while 1% is death rate, resulting in net growth rate of 1.2%). India=1.4%, USA=0.8%, China=0.5% population growth. In year 2016 in USA, there were 4M births a year (40% to unmarried women) and 2.7M deaths (0.6M due to heart disease, 0.6M due to cancer, 0.2M accidents, life expectancy=79 years) resulting in net birth rate of 1.3M or 0.4% per year. Net immigration is 1.2M or 0.4% per year (green card=0.5M, other visas/illegal=0.7M), which causes US population to grow by 0.8% (2.5M people per year). By 2058, US population is expected to cross 400M. Population growth is expected to slow down to 1.5M people per year (0.4M will be natural increase, while 1.1M would be due to net immigration). So, net immigration is going to be the driving force for population growth in USA. If not for population growth from immigration, GDP growth would be less than half of what it is. So, USA does need people from other countries to keep on coming in for it's own good.

One noteworthy data from deaths based on age shows that 80% of the deaths occurred due to age age. For 2017, 0.9M deaths occurred at age of 85+. Other 0.3M for ages 80-84. In fact, 2.1M or 75% of the deaths occurred for people aged 65+, Since 60M people are age > 60 yrs in US, this implies that 5% of the people in this age group die every year. Deaths in age groups < 60 years is less than 0.2%.

https://www.cdc.gov/nchs/data/nvsr/nvsr68/nvsr68_09-508.pdf

Population growth via immigration based on country of birth.

This link gives important stats for year 2009: https://www.migrationpolicy.org/article/frequently-requested-statistics-immigrants-and-immigration-united-states-1/. US immigrant (foreign born irrespective of current citizenship) population as of 2009 was 39M (mexicans=10M, Phillipines=1.8M, India=1.7M, China=1.6M, Vietnam=1.2M). It increases by 0.5M or 1.5% every year (In 2016, it was 44M). In 2016, 1.5M new immigrants came to USA (Indians=180K, Chinese=160K, Mexicans=150K, Filipino=50K). Till 1970, there were only 10M immigrants, but started increasing rapidly since then, because immigration related strict laws (immigration and nationality act) were amended to more favorable laws in 1965 (immigration capped to 0.3M per year), particularly for immigration from Asia. Of the 39M immigrants, 17M became US citizens, 13M permanent residents, 11M unauthorized immigrants (not sure?) and remainder on various visas. Of these 39M, 90% of them came to USA on business/tourist visa, while 2M came on work-visa, while 1M on student visa.  Nearly 6M visa were issued in 2009, of which 4M were business/tourist visa, 0.4M were student visa(F-1, F-2, F-3) and their family, and 0.4M were exchange visitors (J-1, J-2) and their families. As can be seen that most of the people who came to USA on visa went back, since immigrant population on average goes up by 1M every year. About 0.5M new immigrants come thru green card and others thru illegal immigration, so only very few of these 6M people actually stay in US permananetly. These 39M immigrants make up 26M households. In 2009, nearly 1.2M immigrants become permananet residents of which 0.7M are existing immigrants, while 0.5M are new arrivals to USA. Also, 0.8M immigrants became US citizens (Mexico=112K, India=52K, Phillipines=39K, China=37K, Vietnam=31K)

Asian Indians in USA:

basically 1 out of 100 people is USA is an Asian Indian (defined as one of Indian origin => He could either be one who is born here, but has ancestors from India, or one who has been born in Indian, but is currently in USA).

Following wikipedia link gives useful info: http://en.wikipedia.org/wiki/Indian_American#Statistics_on_Indians_in_the_U.S.

Of these 2.9 million indians in USA, 1.7M are India born who either became naturalized citizens, permanent residents (0.5M) or just temporary residents on various student or work visas. Remaining 1.2M are USA born, who are USA citizens by birth.

As of 2012, There were total 13.3M permanent residents in USA, of which 0.5M were Indians (Mexico=3.3M, China=0.6M, Phillipines=0.6M). Also, there were 0.4M H1B Visa holders in USA of which 0.3M were Indians (China=50K, no ther country more than 5K)

Cities:

There are only 10 cities in USA with population > 1M (i.e 10 lakh people). In contrast, there are 50 cities in India with population > 1M

Link: https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population

1M + : 10 cities

400K+ : 50 cities

200K+: 100 cities

100K+: 300 cities (of these about 70 are from california, 40 from texas and 20 from Florida. None of other states have more than 10 cities in top 300 cities)

About 50M lived in top 50 cities. Cities by themselves do not give an idea of size of that area, as any city area is fixed, so as people start moving into the city, the city expands, but the expanded areas become new cities. As the original city itself doesn't grow in area, it's population kind of maxes out after a while, and adjoining cities keep on taking extra influx of people. A better stat is MSA (metropolitan statistical area). It gives better picture how the city population and area is growing. Link below shows that just 5 MSA comprised of 50M people. Top 15 MSA comprised of over 100M people, while more than half of US population is in top 35 MSA. About 70% of population is in top 100 MSA, and 85% in all 383 MSA.

Link: https://en.wikipedia.org/wiki/List_of_metropolitan_statistical_areas

Historically, New York, Los Angeles and Chicago has always been the top 3 most populated cities as well as MSA in USA. These MSA comprise about 15% of USA population (while these top 3 cities comprise only about 5% of population, so we can see that MSA are more relevant). Area wise also these are huge MSA. Since top 100 MSA are where most of the high paying jobs and industries are, these are essentially where people mostly live. These are the MSA where housing prices matter, since these area are restricted by available space to build housing which is close by to work (within 30 min of commute time). Outside of these 100 MSA, house prices should pretty much be same as the cost to build house, since there's abundant land. Land as a percentage of house sale price is less than 10% for MSA which are not in top 100.

Employment:

Population and employment figures from year 2010: (taken from www.census.gov, and www.bls.gov)

USA population is 310M, out of which 155M are in the civilian non-farm workforce (1.5M are in military workforce, while 3M farm for a living. If we include total workforce is 160M). About 10M out of this civilian workforce is unemployed, so 145M people are employed. Simple calculation for population is that there are roughly 20M people in each age group of 5 year increment, starting from 0yrs upto the age of 60 yrs. So, population in 0-60yrs is 20M*60/5=240M ppl (In reality, it's 250M). Remaining 60M are over the age of 60 yrs. Population breakup by age is:

< 5yrs: 20M. Out of these, 6M goto some form of preschool.

5-17yrs: 54M. Out of these 55M are enrolled in kindergarten, elementary or high school (50M in public and 5M in private). So, all of the population is getting primary education. That is reflected in the workforce number, where 87% of the workforce has a high school degree. That implies, about 10% of these 55M kids drop out of high school or even before, without getting a high school degree. More details here: https://nces.ed.gov/programs/digest/d17/tables/dt17_105.20.asp?current=yes . Total student enrollment is 76M, of which 55M are in schools (38M in elementary/middle school, 16M in high school), 21M are in colleges (18M undergrads, and 3M grads). 14M high school kids are in this age group, while none of college kids are in this age group.

18-24yrs: 30M. Out of these, 14M are enrolled in high school or college (school=1.5M, undergrad/grad college=12.5M). So, more than half of this age group people are not getting any college education. Of the ones that are enrolled, half of them drop out without geting a 4 year degree. So, only 25% of the workforce has a 4 year college degree or higher, which comes close with the workforce numbers of 30% workforce having a 4 yr college degree . Note, many of the people in higher age group, also get a 4 yr college degree later in their life, so the workforce has a slightly higher number for people with 4 yr college degree. Also, out of the remaining 75%, about 2/3rd of them enter the workforce. So, out of 22M who don't complete college, about 15M get into workforce and start earning money.

25-59yrs: 145M. 8M of people here are still students in colleges. Most of the people in this group comprise the workforce of 155M. BLS figures show a partcipation rate of 85%, so about 115M of the workforce is comprised of this group. Looking at this number, only 20M (145M-8M-115M) of this age group is not employed. Some of them are stay at home mom/dad, some may be involved in farming, while some may be out of labor force all together because of disabilities, etc, while some are simply unemployed because no one is hiring them.

> 60yrs: 60M. about 25% of the people in this age group still work, implying about 15M of the workforce comes from this age group. Note that full retirement benefits (social security benefits) don't start until the age of 67yrs, so many people keep on working till age 65. In fact, in the age group of 60-65yrs, of a population of 17M, 11M keep on working. Remaining 4M of the workforce comes from people over age 65.

So, based on this, we see that abou 80% of the workforce of 155M comes from age group 25-60yrs, and only 30% of the workforce has a 4 yr college degree. Kids+students are about 100M, while old retired people are about 45M. Out of remaining 165M people who are eligible to work, 150M people are working (including military and farm jobs). Only 15M are not working (5M due to choice or disability, while 10M due to not able to find a job). One interesting fact is that females in colleges outnumber males (12M vs 9M).

Salary:

Now, lets see at the occupation of this workforce and their median salary. This is a good link at bls website: http://www.bls.gov/cps/cpsaat39.htm (NOTE: this only shows earnings for people making more than certain threshold/year, since for lower earnings, data may get unreliable). Unemployment rate for people with bachelor's or higher degree is very low (2.5% in 2017), while those who didn't complete high school was much higher at 13%.

Now, we look at what profession are these people employed in.

Highest paying occupations: about 17M people are in this group making > $100K/year. These are top 10% of working population. Highest paid jobs are doctors, lawyers and CEOs most of whom are in top 1%.

  1. Medical Professions: These include anyone working in Medical field, as doctors, nurses, therapists, etc. NOTE that doctors are the only ones in this field who make salaries in top 1%. Others make much lower salaries.
    • Doctors: Total 1M physicians, surgeons, dentists, pharmicists, dietician or related occupation. Doctors are the highest paid of any profession in USA. Median salary=$200K for salaried doctors, while self employed or those working in speciality fields earn about $400K in median salary. In fact, 9 of top 10 of the highest paying jobs in the USA are all in the field of health care profession (i.e doctors).
    • Therapists: 0.5M therapists, as speech therapist, physical therapist, respiratory therapist, etc. Median salary=$100K.
    • Nurses: 2.5M. Salary=$100K
    • Technologists, technicians, vocational nurses: 2M. Decent salary > $70K.
  2. CEOs: 1M CEOs, including those who have their own business. median salary = $200K. If we only look at large public or private companies, there might be only 20K CEOs, and their median salary is >$500K.
  3. Legal Professions: This includes lawyers and paralegals, as well as people who have their own law firm. They are usually very highly paid, and lawyers in top companies as well in good legal firms usually end up in top 1% of the population earning wise.
    • Lawyers: 1.2M lawyers (according to American Bar Association). Median salary=$200K.
    • Paralegal, legal assistants and legal support workers: Total=0.5M.
  4. Technology Profession: Here, we include all tech field profession mostly Engineers and computer programmers, etc. Salaries for these people is also pretty high, and they usually start in the top 10%-20% of earners, and by mid career are usually in top 3%-5% of earners.
    • Computers: 3M in the field of software programming, IT, DB, Web developer, etc.
    • Engineering: Total = 2M. Electrical engineers, Mechanical engineers and civil engineers are 0.3M each, while other fields as aerospace, chemical, nuclear, materials, industrial, etc comprise remaining 2M. Engineering jobs are the highest paid jobs coming out of college across all disciplines.
    • Engineering Technicians/drafters: 0.5M
  5. Life, physical, and social science: 1M. Surprisingly salaries are at par with engineering jobs, at about $100K. These mostly include phychologist, scientists, economists, Miscellaneous life, physical, and social science technicians, etc.
  6. Arts, design, entertainment, sports, and media occupations: 1.5M. Mostly includes artists, actors, writers, directors, dancers, sportsman, entertainers, photographers, media workers, etc. avg salary is about $100K.

Lowest paying occupations: about 15M people are in this group making < $40K/year ($20/hr *40 hrs/week *50 weeks = $40K). When you hear headlines of people losing jobs, these are the people who lose their jobs. So, unempoyment rate is mostly affected by people in this category. They are bottom 10% of working population.

  1. Agricultural/farm workers: Total=1M. lowest pay of all occupations at < $10/hr. Doesn't require any education. They usually work on farms of other people.
  2. Food preparation and serving related occupations: Total=4M. Mostly employed in small businesses, as waiters, cooks, attendants, etc.
  3. Personal care and service occupations: Total=2M. Mostly includes hair dressers, child care, animal trainers, fitness workers, etc.
  4. Building and grounds cleaning and maintenance occupations: Total=4M. Mostly includes janitors, maids, ground maintainence, etc. Very low pay at < $10/hr.
  5. Protective services: Total=1M (excludes govt employees as police, fire fighters, etc). Includes security guards, life guards, etc.
  6. Healthcare support occupations: Total=2M. Includes all support staff, assistants at hospitals, etc.

Middle paying occupations: remaining 120M people are in this group making $40K - $100K. These are the middle 80% of the working population and comprise everyone else. This is WIP? FIXME?

  1. Teachers: We include anyone in Teaching profession here. They are teaxhers in schools, 2 yr colleges, as well as professors in universities. Most of the professors in top universities get paid pretty well to be in the top 10% of population. Howver, they are still significantly under paid compared to their colleagues at private corporations.
    • Schools: 3.7M teachers in elementary/high schools (3.2M in public schools and 0.5M in private schools). Assuming 50M students in public schools, that's a ratio of 1:16 for teacher:student. For private schools, it's 0.5M teachers for 5M students implying a ratio of 1:10. About 100K public schools (there are about 14K public school district, meaning avg of 7 schools per ISD (independent school district) and 35K private schools. About $650B was spent in public schools in 2018, implying per student expenditure of $12K/year. If we add up private schools, about $1T is spent in schools every year.
    • Colleges: Assuming avg college fees of $25K/year, and 20M students, about $0.5T is spent in colleges every year.  There are about 3000 4 yr institutions, and 1700 2 year institutions. https://nces.ed.gov/fastfacts/display.asp?id=372.
  2. Misc: Many more professions here. I'll list some more common ones here later