Details

Published: Monday, 08 July 2019 19:28

Hits: 1101

When building any website, we need to know about 2 things:

1. language to code your webpage in = usually php

2. language to build a database of objects = usually mysql (mysql is really a client/server software, which uses SQL language to allow client to talk to server)

Think of database as a big file that stores all your data. Usually this data gets very big (i.e a website that has a list of all items, their price, description, etc). If we store all this data in 1 big file, then just reading or writing to this massive file every second will be very slow. This is because appr location in file needs to be searched for and then data written over there.  Also not feasible, since updates will be happening to this file like every micro second, whenever a customer places an order, or prices change on items, etc. This will make operations to file very slow. So, a special software comes to our rescue called database management software (DBMS). DBMS still store data as one big file (we have options to break it in smaller files), but their real value comes from their ability to be able to do quick searches, maintain transaction atomicity, prevent datafile from getting corrupted, etc.

There are many varieties of DBMS. One of them particularly popular is RDBMS (Relational DBMS). This is called relational database because all the data is stored into different tables and relations are established using primary keys or other keys known as Foreign Keys.

In DBMS terminology, database refers to a collection of tables with relation b/w them. Tables are just a simple data matrix (like excel sheet) with rows and columns.

MySQL: MySQL is a fast, easy-to-use RDBMS. It's open source, and can support large databases (upto millions of Terabytes).. SQL is a data language, and MySQL uses that language to access the database it stores. MySQL cmds can be accessed via functions in PhP, Perl, Python, etc, so it's really easy to work with MySQL, w/o even explicitly learning SQL cmds. MySQL is wriiten as capital M, S, Q, L, and small y, but below we'll just write it as mysql frequently to promote my laziness.

MySQL pgm has client server architecture. This is very common for pgms, which don't always run on local m/c, and can be accessed from anywhere. MySQL pgm has 2 main pgms => server pgm and client pgm. client pgm runs on local m/c when you type "mysql" on unix promot. Server pgm (called mysqld) runs automatically whenever the m/c is started (provided mysql is installed). As you can see server and client pgm can run on any m/c, and don't have to be the same m/c.

MySQL: good doc here: https://www.tutorialspoint.com/mysql/mysql-quick-guide.htm

One other more comprehensive place is: http://www.mysqltutorial.org/

Offical mysql website also has decent tutorials: https://dev.mysql.com/

Install MySQL: The below link on mysql offical website shows steps in detail on how to install it for apt, yum, etc.

https://dev.mysql.com/downloads/

These are the cmds for installation on linux mint (or any linux derivative using apt):

1. sudo apt-get update

2. sudo apt-getinstall mysql-server => This installs mysql server, mysql client and many other files. Here it installs distribution=5.7, which may not be the latest (there are additional cmds listed which will get you the latest). We do not need to install both client and server, as many times, we don't even need client/server on same m/c. So, we can choose and install only the part that we need. For our purpose, we install both client and server on our local m/c.

After installation, mysql server automatically starts. These are the 2 main binaries that get installed:

A. mysqld => server pgm that contains database. It manages access to the actual databases on disk or in the memory. Binary is /usr/sbin/mysqld. When MySQL server starts, it listens for network connections from client programs and manages access to databases on behalf of those clients. If we see list of processes running (using ps -ef), we will see mysqld running

ex:  /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid => "ps -ef | grep mysqld" shows mysqld daemon running with shown args. mysqld has many options that can be specified at startup.

B. mysql   => client pgm (cmd line) that connects to mysql server to manipulate the information in the databases that the server manages. Binary is /usr/bin/mysql. This doesn't start automatically. This is the pgm that we invoke all the time (we won't talk about mysql server pgm anymore, as it's not relevant to most users). Whenever we say "mysql", we almost always means mysql client pgm.

After installation, verify that mysql is installed:

mysql --version => shows mysql client version = mysql  Ver 14.14 Distrib 5.7.27, for Linux (x86_64)

mysqld --version => shows mysqld client version = mysqld  Ver 5.7.27-0ubuntu0.18.04.1 for Linux on x86_64 ((Ubuntu))

mysqladmin --version => shows "mysql admin" script version = mysqladmin  Ver 8.42 Distrib 5.7.27, for Linux on x86_64. mysqladmin is additional pgm included with mysql package to simplify admin tasks.

mysql --help => shows all options for mysql client pgm

MySQL server allows only certain users on certain machines to connect to it for security purposes (just like ssh, etc). It has a list of usernames and passwords stored in one of it's table, along with the name of host machines from where they can connect, and only those users are allowed to connect. Else we get "Error: 28000: access denied" from the server. During mysql installation, a "root" user account with no password on host "localhost" is defined and added to this table.This means a user  "root" on local machine (or on ip addr: 127.0.0.1) can connect to mysql server on that machine using mysql client. Any other user from any other m/c is not allowed to connect to mysql server. So, if we want to run "mysql" client pgm on the same m/c to connect to mysql server, we need to change user to "root" on our machine, meaning we need to use a "sudo" cmd preceeding mysql cmds (or run "sudo -i" to change to root, and then use cmds w/o sudo. su wouldn't work as root a/c is disabled by default in many linux distro for security reasons),

So now we know, that anyone logged in the m/c as root, or using cmds with sudo in front (which gives them same rights as root) can connect to mysql server. mysql server by defaut doesn't require a separate password for root user  to connect to it. This is risky. To fix this, we'll set  password for root (for mysql server user "root").

If we look in /etc/passwd file, we will see list of all users on that machine. We'll see user "root" and user "LOGIN_USER", along with other users. "LOGIN_USER" here is the name of username with which we log into our linux machine (for ex: for username "Rakesh", LOGIN_USER is Rakesh, so we'll see username "root" and "Rakesh" in passwd file). After installing mysql, we also see a new user "mysql" added to this file, as mysql pre-install script adds a user "mysql" using "adduser" cmd.

The mysql.user grant table defines the initial MySQL user account and its access privileges. Installation of MySQL creates only a 'root'@'localhost' superuser account that has all privileges and can do anything. If the root account has an empty password (which is the default), your MySQL installation is unprotected: Anyone can connect to the MySQL server as root without a password and be granted all privileges (although only on local m/c). So, we assign a password to "root" user using any of 2 scripts below, before invoking mysql client. NOTE: we use "sudo" before each cmd, since only "root" user is allowed connecting to mysql server.

1. sudo mysql_secure_installation => this brings up the script to tighten up security for mysql. Most important thing to enter here is your mysql client "root" account's new password.  More details here: https://www.computerbeginnersguides.com/blog/2017/08/02/install-and-configure-mysql-server-5-7-on-linux-mint-18-2/

2. sudo mysqladmin -u root password "new_password"; => this gets new passowrd assigned to "root" a/c on mysql server

cmds to start/stop mysql server: (we use service cmds here, check in linux server notes). All cmds preceeded by sudo.

1. sudo service mysql status => shows mysql is active (running). As mysql automatically starts running on installation

2. sudo service mysql stop => stops mysql server

3. sudo service mysql start => starts mysql server. For restart, we cn also use "mysql restart"

starting MySQL client:

You can connect to your MySQL server through the MySQL client by using the mysql command on cmd line. We have to specify the sql server m/c ip addr (or name), user name who is connecting to it and password for that user. If you haven't assigned any password to "root" account on mysql server, then you can connect to mysql server w/o giving any password as by default it will be set as blank. Also, we'll need to connect as user "root" since no other user is allowed to connect.

> sudo mysql => typing this makes mysql monitor for client appear, and it connects to mysql server locally (Since default host name is localhost). The default user name is unix login user, but by typing "sudo", we become "root" for purpose of executing that cmd. This logins as user "root@localhost", as sudo changes user to "root". IF we don't use "sudo" or are not "root" user, well get access denied error, as user will be unix login user, who doesn't have access rights for mysql server.

cmd line options for mysql client:  Most of the cmd line options can be given in short form as "-o <NAME>" or in long form as "--option=<NAME>". We can also specify all these options in an option file instead of typing them on cmd line.

-h <hostname> => specifies host name, default is localhost. Can also be written as --host=<hostname>

-P <port_number> => specifies which port on the host to connect to. NOTE: capital "P" Mysql server pgm is bound to TCP port number 3306 by default. Since MySQL server can connect to any remote client, it has to be assigned a port number from which it grabs all it's input. This is true for TCP/IP protocol which works that way. It can be any one of 65K ports assigned for connections using TCP/IP. This port is just a chunk of memory (buffer) where incoming data from internet connection is dumped so that specific pgms continuously monitoring this port can start consuming data in this buffer. This port buffer will have additional info on which ip addr the data is coming from, user name, permissions, etc for mysql server to use. That is why when we access mysql server locally, we would expect it to be coming from an ip addr 127.0.0.1(i.e  "localhost"), from port 3306. However in reality, localhost doesn't use TCP/IP protocol, and instead uses unix socket file to transfer data. Socket file are special files used for Inter process communication by the OS. API for unix socket is similar to that of Internet socket, but all communication occurs entirely within kernel, and filenames are used as addr instead of IP address (since communication is all local). So, -P has no meaning for localhost and is ignored.

-u <username> => specifies username, default is unix login user

-p<passwd> => specifies password for that user, NOTE: there is no space b/w -p and password. Also, if we use just -p with no password, then mysql prompts for password on next line, which is safer, since password is not visible in plain text. Also, all processes along with their options can be seen by typing "ps -ef" by any user on that m/c, so anyone can see user's password, if password was entered on cmd line. So, always use "-p" with no password. 

<db_name> => we can optionally specify name of database to open (with no - option) that we want to open. If we don't specify database name, we connect to sql server but can't access any database, until we select a database using "USE" SQL cmd on sql client prompt.

ex:

> mysql -h example.com -u rajat -p => here we connect as user "rajat" on host "example.com on default port 3306. If rajat is not an autorized user in mysql server table on that host, then he can't connect. Since no password provided after -p, then password is prompted on net line.

Enter password:*******

mysql client prompt appears at this time, and we can start typing mysql cmds.

 mysql dir structure: After installation, we'll have many dir with mysql files in them. Details of source code and dir structure is here (very good descr, should read this): https://dev.mysql.com/doc/internals/en/

Here is how the flow works:

1. User invokes mysql client by typing "mysql" with appr options. client connects to the server, and does initial handshake. Encyption and authentication methods are established (SSL can be used for both encryption and authentication).
   • In a classic client/server, the server has a main thread which is always listening for incoming requests from new clients. Once it receives such a request, it assigns resources which will be exclusive to that client. In particular, the main thread will spawn a new thread just to handle the connection. Then the main server will loop and listen for new connections.
   • MySQL has a great variety of mutexes (mutual exclusion object) that it uses to keep actions of all the threads from conflicting with each other.
   • For each thread, there is an infinite loop. The loop repeatedly gets and does commands received from sql client. When it ends, the connection closes. At that point, the thread will end and the resources for it will be deallocated.  
   • All communication b/w client/server is done via packets. Packets are formatted messages that client/server send over tcp/ip lines. Packets will have a header, an identifier, and a length, followed by the message contents.  
2. User types any SQL cmd on mysql cmd prompt. client performs initial checks. Source code for this sits in client dir of mysql s/w. As an ex, we enter SQL cmd INSERT =>  mysql> INSERT ,,,
3. The mysql client then passes this edited/checked cmd over TCP/IP to sql server @ip_address and to default port for sql server. If server is local, then ip addr is 127.0.0.1, and file socket used. These cmds are passed in form of  packets.
4. Then via the vio routines (in C), the server grabs this cmd, Low level mysql routines are called to parse sql (source code in sql dir), and doing what's necessary (Source code in mysys dir)
5. Finally, one of the ha (handler) programs in the sql directory will dispatch to an appropriate handler for storage. Depending on the storage engine, handler may be in myisam or Innodb dir.

Few of the important dir are:

1. /usr/bin, /usr/sbin => These directories contain mysql/mysqld binaries

1. /etc/mysql/ => This has many configuration files. Most important is database server configuration file: /etc/mysql/mysql.conf.d/mysqld.cnf. This file has settings for where all log files, data files etc are stored. Data file for our database is stored as 1 big file here:  /var/lib/mysql/ibdata1

2. /var/lib/mysql => this dir has all logs, data, etc. We see ibdata1, which is the some data file that stores databases. More data files would be named ibdata2, ibdata3, etc. Note sure what they store ?? FIXME??. Inside this dir, we see subdir for all databases (as mysql, sys, etc). If we create a new database, a new dir with that name gets created with file db.opt in it. It is a 2 line file, with charset defined for that db. When we create new tables using "create table customer" then 2 new files for that database get created - customer.frm and customer.ibd. These are both binary files.Each table has it's own .frm and .ibd files, along with lot of other files. In other database dir as "mysql", we see lot of other files.

.frm => every MySQL table you create is represented on disk by a .frm file that describes the table's format (that is, the table definition). We can do hexdump to see contents of this file. It shows header which stores various info about the table.

hexdump -v -C customer.frm

.ibd => These store actual data, i.e table values inserted. Data is stored in compact byte form, Each data assigned storage space in file based on it's data type. This is much larger file, as it stores data.

SQL cmds:

We can type SQL cmds directly on mySQL client prompt, or put all the cmds in a file, and then run that file on mySQL client.

• Each SQL cmd needs to be terminated by ";".
• SQL is not case sensitive, but convention is to hav capital letters for SQL cmd and small letter for arguments.
• 3 style of comments in SQL. # and -- treats anything from # or -- to end of line as a comment. -- needs to be followed by atleast one whitespace for it to be recognized as a valid comment. C style /* ... */ is used for in line as well as multiline comment.

1. Databases: A mysql server can keep several databases for different websites or projects, all in same server. This is accomplished by giving separate names to databases. Each of these databases can have multiple tables to store the actual data. The complex security mechanism ensures that individual databases and tables can be accessed only by authorized users. That is why we have to provide a username when starting mysql client, so that mysql client can check which databases and tables are allowed to be accessed by that user. Database related cmds:

- SHOW DATABASES; => shows all databases.By default, mutiple databases are already there. "mysql" is one of the databases that is automatically built on installation that stores user info.

- USE mysql; => this selects a database to work with. This cmd is essential to start working with database, else no database is selected, and so no database related SQL cmds will work. We can also select a database when connecting to mysql client via -D option, i.e mysql -u robert -D mydb -p => selects mydb database
 
- SELECT DATABASE(); => This shows all selected databases.

- CREATE DATABASE test_db; => creates a new database named test_db, This creates a dir named test_db in /var/lib/mysql. Inside test_db will be a *.opt file which stores various configuration for this db. We can only select 1 database at a time, so "USE test_db;" will select this database instead of any previously selected one. In mysql, SCHEMA is the synonym for DATABASE, so they can be used interchangeably.

- DROP DATABASE test_db; This deletes the said db.

 

2.Tables: Tables are most important part of a database. Here it is where all all the data in a database is finally stored. There are various table types also called storage engines that you can choose from. These storage engines are internal way used by a database server to store data. Uuually we should not care about storage engine, as we do not need to change it. The default storage engine from version version 5.5 an above is InnoDB, while before that it was MyISAM. Besides these, there are many other storage engines as MERGE, ARCHIVE, CSV, etc.

A. InnoDB: The InnoDB tables fully support ACID-compliant and transactions. They are also optimal for performance. InnoDB table supports foreign keys, commit, rollback, roll-forward operations. The size of an InnoDB table can be up to 64TB.

B. MyISAM: The MyISAM tables are optimized for compression and speed. However, they are not transaction safe.

Tables store data as a table with rows and columns. We specify how many columns each row has, what is the data type that each entry in a particular column, and that's it. The data types are of 3 categories:

I. Numeric: This includes INT (signed or unsigned upto 11 digits), BIGINT (upto 20 digits), FLOAT, SMALLINT, etc.

II. String: This includes BLOB or TEXT (max of 65K char), VARCHAR (from 1 to 256 char, num of char needs to be specified in brackets as VARCHAR(5) means 5 char), CHAR (CHAR is a string with default char length as 1, but can be defined as CHAR(5) indicating it can have up to 5 char), LONGBLOB or LONGTEXT, TINYBLOB or TINYTEXT, ENUM, etc. The difference b/w BLOB and TEXT is that sort and comparison on BLOB data is case sensitive, but not so for TEXT. BLOB is Binary large object and is used to store large amount of binary data too. ENUM is enumeration or a fancy term for list. Just as in other languages, it allows only those values put in the list to be allowed as valid. ex: ENUM( 'ABC', '12') will allow only ABC, 123 and NULL as valid values for that field.

III. DATE and TIME: This includes DATE (stored in YYYY-MM-DD format), TIME (stored in HH:MM:SS format), DATETIME (stroed in YYYY-MM-DD HH:MM:SS format,)

NOTE: NULL is a special data type which represents "nothing", and it can be applied to any of 3 data types above, whenever we do not want to store a value. Any of above 3 data types above can have NULL value.

These are few table related cmds.

1.CREATE TABLE table1( ... ); => This creates tables with specified columns. Each column has data name, data type and optional data attributes. ex:

CREATE TABLE customer (

Idx INT AUTO_INCREMENT, #This is 1st col. It has field called "Idx" which is of type Integer, and it's value is auto incremented, whenever a new row is created. AUTO_INCREMENT is an optional attribute. Field Attribute AUTO_INCREMENT tells MySQL to go ahead and add the next available number to the id field.

name VARCHAR(236) NOT NULL, #VARCHAR(236) says it can have upto 236 char. "NOT NULL" attr specifically states that "name" is not allowed to have "NULL" value, so if we try to create name with "NULL" value, mysql will raise error.

start_date DATE,

description TEXT,

PRIMARY KEY (Idx) #NOTE: no comma for last field. Keyword PRIMARY KEY is used to define a column as a primary key. You can use multiple columns separated by a comma to define a primary key.
)  ENGINE=INNODB; #Engine name is optional. NOTE: semicolon at end of this line to indicate end of SQL stmt. So, above cmd is 1 long cmd spread on multiple lines for clarity.

2. INSERT INTO table1 (field1, filed2, ...) VALUES (value1, value2, ...); #after creating tables, we use this cmd to insert values into fields of table. We use this cmd repeatedly to insert as many rows as we want to the table. ex:

INSERT INTO customer (description, name, start_date) #Idx is not inserted into table, as Idx has AUTO_INCREMENT attr, which means it's incremented by 1 and assigned to "Idx" for this row, whenever we insert a new row.

VALUES ("Learn PHP", "John Poul", NOW()); #Here NOW() is a function which returns date.

3.UPDATE, DELETE, etc used to update/delete values in fields.

4. ALTER TABLE => Just as we can add, delete. modify values in rows, we can change columns for a given table. i.e add columns, remove columns using this cmd.

5. SHOW TABLES; # shows all tables in selected database.

6. SHOW COLUMNS from table_name; #This shows all the info for all columsn from the given table.

The 3 cmds, SHOW DATABSES, SHOW TABLES and SHOW COLUMNS can give all the info about whatever is present in any data on a sql server. Databases have tables, and tables have columns. Columns are the lowest hier that store the data. To prevent anyone from accessing all this info, mysql provides privileges at finer level up to the table. So, only privileged users can access a certain database and certain table, thus providng security to data stored. Database "mysql" stored by default in mysql server has one of the tables as "user". If we list columns of user, we will see these privileges.

ex: SHOW COLUMNS FROM user; -- Shows all colums as User, Host,  various privileges, etc (total of 45 columns) from table user in database named mysql.

+------------------------+-----------------------------------+------+-----+-----------------------+-------+
| Field                  | Type                              | Null | Key | Default               | Extra |
+------------------------+-----------------------------------+------+-----+-----------------------+-------+
| Host                   | char(60)                          | NO   | PRI |                       |       |
| User                   | char(32)                          | NO   | PRI |                       |       |
| Select_priv       | enum('N','Y')                   | NO   |         | N                   |       |

7. SHOW VARIABLES; => shows all variables and their values in use. variables are current database configuration settings (set by default or modified by user). It shows that default port for mysql server pgm running is 3306.

8.SELECT => This is the most used cmd to select fields from a given table. We can use a lot of complicated search criteria (by using WHERE clause) and get only the rows and columns meeting that search criteria.

ex: SELECT * FROM user; => This shows all fields (columns) for all rows from table "user" in database "mysql" (assuming mysql db was selected before). * denotes "all".
ex: SELECT name, salary FROM tutorials_tbl WHERE tutorial_author = 'Sanjay'; => HEre we select 2 fields (columns) from table "tutorials_tbl". WHERE clause adds extra filtering criteria, saying the field "tutorial_author" needs to be "Sanjay". So, only those rows will be selected which match the WHERE clause, and only 2 columns "name" and "salary" would be displayed. If * was used, all columns would be displayed. Many other keywords as AND, OR, <, !=, etc available for doing a match.

 3. Misc cmd => There are tons of other cmds for doing doing lot of other things with databases.

Adding new user to db:

So far we did everything on mysql db as "root" user. However, we'll want to allow valid users besides root to access mysql db. This can be done via creating a user, and then granting it privileges, Done via following cmds:

mysql > CREATE USER 'ashish'@'localhost' IDENTIFIED BY 'password'; #here, we created a new user 'ashish', but no privileges assigned, so we can't login with this user yet

mysql> GRANT ALL PRIVILEGES ON * . * TO 'ashish'@'localhost'; #grant all privileges to this new user when logged from localhost. First * refers to database while second * refers to  table. So, here we grant all privileges to user ashish on all db and all tables. Not a safe thing to do, as now user 'ashish' can modify anything in any db.                                          

mysql> FLUSH PRIVILEGES; #To reload all the privileges so they take effect immediately

 mysql> use mysql; #To check ifnew user has been added, we check in "mysql" database, and "user" table.

mysql> select * from user; # this shows a row for new user "ashish"

support for mysql in other languages:

We can connect to mysql server via mysql client by invoking it from terminal. However, the real value in mysql db is when we can connect to the db from within some other pgm. For this reason, popular web languages as php, python, perl, etc provide functions to connect to mysql server, rd/wrt to db, and basically do everything from within the language. php language provides extensive support for mysql. Details of this approach are explained in php tutorial.

Migration of Database: Copying/Moving MySQL Database from 1 server to other is not difficult. We know that database contents are stored in *.idb file (for InnoDB). However, moving these files by themselves is not going to work, as they rely on *.frm files and host of other files. Also, if the storage engine is different, then a conversion would be required from one format to other. To prevent users from all this hassle, mysql provides various utilities to copy/move the database.

The mysqldump program is used to copy or back up tables and databases. It can write the table output either as a Raw Datafile or as a set of INSERT statements that recreate the records in the table.

1. As raw datafile: ex:

> mysqldump -u root -p --no-create-info  --tab=/tmp/dir1 tutorials_DB tutorials_TABLE => this dumps table=tutorials_TABLE from database=tutorials_DB to file in /tmp/dir1 dir

2. in sql format: we can also dump selected tables or entire database in sql format in a separate file. Here, we write out output file as series of sql CREATE/INSERT cmds, that will recreate the original table.

> mysqldump -u root -p tutorials_DB tutorials_TABLE > dump.txt => same as above except that --tab option is not used. dump.txt is a regular sql file, that has "CREATE TABLE tutorials_TABLE" and "INSERT INTO tutorials_TABLE". To dump multiple tables from the database, provide name of each table separated by space (i.e tutorials_DB tutorials_TABLE1  tutorials_TABLE2 > dump.txt)

> mysqldump -u root -p TUTORIALS_DB > database_dump.txt  => this dumps entire database instead of tables

So, to move copy database from 1 server to other, use these 3 cmds below:

• mysqldump -u root -p MY_OLD_DB > database_dump.sql => dump all database in this file on old server from where you want to transfer
• mysqladmin -u root -p create MY_NEW_DB => create new database named MY_NEW_DB on new server where you want to transfer files to
• mysql -u root -p MY_NEW_DB < database_dump.sql =>Now run the dumped sql file on this new DB, which will create/insert all tables into this new db, using cmds from *.sql file.
  
   
  
   

Details

Published: Tuesday, 28 May 2019 13:45

Hits: 1853

iverilog - open source verilog simulator:

Many open source simulators available. We'll talk mainly about icarus as it's most widely used.

Verilator: commercial quality open source simulator. It's verilog to C translator. Translates Verilog/SV into C++/SystemC code resulting in very fast sim times. Verilator is 100X faster than interpreted verilog simulators as Icarus verilog. More info here: https://www.veripool.org/wiki/verilator

verilog2cpp: another open source simulator. More info here: http://verilog2cpp.sourceforge.net/

icarus verilog: (aka iverilog)

An open source verilog simulator written in C++. Very complicated, supports all verilog constructs. More info: http://iverilog.icarus.com/

Download from here: https://github.com/steveicarus/iverilog

On CentOS, we can download binaries directly, instead of building it. This is on CentOS 7.5.1804 release. Type below cmd on any terminal.
yum install iverilog => It may show some http 404 errors, but eventually will find some other mirrors. On entering "Y" after it asks for confirmation, following gets downloaded and installed. Downloading packages: iverilog-10_2-2.el7.x86_64.rpm           
Installed: iverilog.x86_64 0:10_2-2.el7 Both iverilog and vvp are installedwhich iverilog => shows path as "/usr/bin/iverilog"
which vvp => shows path as "/usr/bin/vvp"There is very good beginner's material here: https://iverilog.fandom.com/wiki/User_GuideLet's copy the hello.v code shown on link above, and run testcase on it: ( > below is the prompt)> iverilog -o hello hello.v
> vvp hello
Hello, World
We can similarly try "counter" example. cp counter.v and counter_tb.v and run below cmds: (finish cmd is needed to get out of simulatoriverilog -o my_design  counter_tb.v counter.v
[root@DESKTOP-G2APOII counter]# vvp my_design
At time                    0, value = xx (x)
At time                   17, value = 00 (0) ....
** VVP Stop(0) **
** Flushing output streams.
** Current simulation time is 168 ticks.
> finish
** Continue **
To get dumpfile from sim, we can add following code anywhere in tb file

initial
 begin
    $dumpfile("test.vcd");
    $dumpvars(0,test);
 end

Now on running cmds again, we will see test.vcd which can be loaded on any waveform viewer.

GTKWave: An open source waveform viewer. More info here: http://gtkwave.sourceforge.net/

Download following software before installing GTKWave.

1. gperf: GNU perfect hash function generator. Link here: https://www.gnu.org/software/gperf/

    download from ftp://ftp.gnu.org/pub/gnu/gperf/
    Download latest version (gperf-3.1.tar.gz). Extract and follow these steps:

    - cd gperf-3.1

    - ./configure

   - make

   - make check => optional. Runs test cases

   - sudo make install => installs it in /usr/local/bin/gperf

 2. gtk: It's used for creating GUI. Link here: https://www.gtk.org/.

  If you go to download page for Linux, you will see latest version as 3.24, while earlier gtk2 version as 2.24. I think you can download either one, however there are lot of dependencies as Glib, Pango, Gdk-Pixbuf, ATK and GObject-Introspection. I tried to install "gtk+-2.24.0", but it fails looking for these dependencies:

configure: error: Package requirements (glib-2.0 >= 2.27.3    atk >= 1.29.2    pango >= 1.20    cairo >= 1.6    gdk-pixbuf-2.0 >= 2.21.0) were not met:

No package 'glib-2.0' found
No package 'atk' found
No package 'pango' found
No package 'cairo' found
No package 'gdk-pixbuf-2.0' found

 I tried to install pango, cairo, glib. glib required meson, which could not be installed due to python 3.5 dependency (even though python3 was installed). Fortunately, CentOS repository has gtk pkg available for installation via yum, so I took that route:

 - yum groupinstall "Development Tools"

- sudo yum install gtk+-devel gtk2-devel => this will download and install gtk with all required dependencies

You will see these 2 pkg installed:

Installed:    gtk+-devel.x86_64 1:1.2.10-77.el7                              gtk2-devel.x86_64 0:2.24.31-1.el7        

Once you have downloaded above gperf and gtk package, we can download GTKWave (gtkwave-3.3.101.tar.gz) at bottom of gtkwave page. Extract it, and in the extracted dir, we'll see a README/INSTALL that lists the steps. Here's the steps:

1. cd gtkwave-3.3.101

2. ./configure --disable-xz => This will error out if the option is not used since xz is not installed.It's a compression software, we'll disable it for now. This is what you see on screen if xz not disabled:

configure: error: LZMA support for VZT is enabled, but xz could not be found.
Please install the xz-devel package, see the http://tukaani.org/xz website, or use the --disable-xz flag.

3. make

4. sudo make install => should get installed in /usr/local/bin/gtkwave

Now run "gtkwave test.vcd". It should bring up gtkwave gui.

NOTE: You get error like this while running gtkwave:

tkwave: error while loading shared libraries: libtcl8.7.so: cannot open shared object file: No such file or directory

This can happen when there are multiple versions of tcl/tk installed on your system. yum install or manual install of tcl/tk may not show any error, but when trying to run a software, we may see errors related to tcl/tk files. Best way is to remove tcl/tk, and start fresh from manual install of tcl/tk (yum may not fix the issue as "yum install" will always show everything uptodate and refuse to update). You may want to use below option to force a particular tcl/tk version to be used in step 2 above, and then continue with step 3 and 4:

./configure --disable-xz --with-tcl=/usr/lib64/tcl8.5 --with-tk=/usr/lib64/tk8.5 => replace version 8.5 with whatever version you have on your system. I used both 8.5 and 8.7. This did not work for me, so I had to reinstall tcl/tk 8.7. After that, it worked, without using these tcl/tk options.

 -----------------

Details

Published: Thursday, 16 May 2019 20:35

Hits: 1587

Meridian RDC:

Meridian RDC is a tool by company called "Real Intent", which does RDC checks. Spyglass by synopsys is other tool that does CDC/RDC checks too. Cadence has their own RDc/CDC checkers.

Install Meridian RDC and then invoke it. Assuming mrdc is the Meridian RDC tool alias:

mrdc -input mrdc.tcl -log mrdc.log -project meridian_project  => we provide i/p file mrdc.tcl which has cmds for all the 4 steps below (instead of -input, we can also use short form -i, and instead of using *.tcl file, we can also use *.rdc file). option -project allows us to dump mrdc internal db into that dir, so that we can restore the old database for debug purpose (tool idebug is used, instead of mrdc. It's invoked as "idebug  -project meridian_project"). option -reset_scenario_mode enables using only reset scenarios instead of groups. So, all cmds use scenarios (scn) in place of group (grp). Total violations in scenario mode is more than in group mode. -reset_scenario_mode will become default in future release.

Steps: All of the tcl cmds are supported in mrdc.

1. design setup => Sets up design for RDC verification. Analyze and elaboarate of RTL done here.


set ri_search_path [list /home/project/ip/src  /home/myblock/rtl ] => optional, setting search path for RTL sources in case we don't provide full path of any rtl file below. There are 100's of meridian var that can be set, look at mrdc manual.
read_liberty [list /home/memory/memory.lib /home/clk_gen/pll.lib ] => optional since we are checking RTL (and not gates), reading in technology specific library cells
analyze -file rtl_file.f -svverilog => analyze all the RTL source files (rtl_file.f is same format as used for any other tool, which has paths of all rtl files, along with any options as "+define+, "-v2k", etc). we can also put options on cmd line as "-svverilog" to enable system verilog
elaborate design_top -black_box [list analog_d2a analog_a2d] => elaborates design with "design_top" as top module. We blackbox blocks that we do not want analyzed or are IP for which there is only verilog model. We can use -auto_black_box to automatically black box any block for which defn is not found, or use -black_box to give list of blocks to black box.

NOTE: elaborate produces internal The database which includes complete description of design. read_design_db cmd can be used to read this database. The name of database can be specified else it's some fixed name (i.e mdrdc.db or something). If we run hierarchical flow for mrdc (discussed later), then we use this elaborate cmd to create db for lower level modules. However, if there are multiple instantiation of this module in top level, then we won't be able to use the same name for different inst of same module. In such scenario, mrdc allows us to create unique db for each such module by having parameter names in name of db. i.e:

elaborate -hier_design_db {dut_block1} => here design db for dut_block1 created with names of all parameter values, i.e for parameter values 1,2,3,4, name=dut_block1-1-2-3-4 where this particular inst of dut_block1 has parameter values 1,2,3,4, similarly for all other parameter values. If -hier_design_db option was not used, then the design db would be named dut_block1 (all such db created under dir meridian_project/dut_block1/dbs/dut_block1-1-2-3-4.db).

read_design_db dut_block1-1-2-3-4 => This cmd used to read the previously created db. So, we don't need to read in all rtl files, we can just read this db for next run. This makes it load design faster. We don't specify dir, as it looks in standard path under project dir (here it's under meridian_project/dut_block1/dbs/)

2. environment setup => optional, needed only if SDC (.sdc) or ENV  (.env) files are already available. SDC is std sdc file, while ENV is native enirnoment file in lieu of sdc. Meridian needs these files to know what signals are clock, primary reset and functional reset. Primary reset are ones that are externally generated, while functional reset are ones that are internally generated (e.g. output of a counter, output of a S/W register, …). Constants should also be provided (i.e jtag_mode is set to 0, etc). Meridian can automatically figure out a lot of this info, and dumps that into a template file for user to review (incase the user doesn't provide the sdc or env file with this info). Any problems found are reported under "SDC_ENV_LINT" Rule group.

There is 1:1 conversion from sdc cmd to env cmd.

SDC: create_clock => ENV: create_clock. create_waveform/create_derived_waveform are used to define a waveform, and then these are used in "create_clock -waveform waveform_1 clk_1" cmd.

SDC: create_generated_clock => ENV: create_clock. create_waveform/create_derived_waveform

SDC: set_case_analysis => ENV: set_constant

SDC: set_input_delay/set_output_delay => ENV: create_input/create_output

ENV: create_reset (no equiv cmd in sdc) => this cmd needed to specify reset. Reset analysis is done in both assertion phase (i.e while reset is active) and de-assertion phase so that initial values are properly set on storage elements (i.e flops, latches, memory) as well as primary inputs for structural or formal analysis.

set_false_path and set_clock_groups are used to determine whether clocks are synchronous or asynchronous.


read_sdc [list /home/mydesign/clock.sdc  /home/mydesign/io.sdc  /home/mydesign/exception.sdc ] => reads in sdc to derive clock information, design boundary relation, and synchronous/asynchronous relationship
#read_env [list /home/mydesign/design_top.env ] => read an existing ENV file to derive design spec. We read either sdc or env file.

 #reset groups => reset group is a set of resets that are activated together during functional operation, along with any mode constraints at the time of activation.  Reset groupings control how checks are created - some checks occur only on resets within a group some checks occur only across groups. A reset domain is a set of flops that become initialized when a "reset group" is asserted. If we don't define reset gropus, then our results will be very noisy, as RDC won't won't know how these resets work with each other. Meridian RDC defines "reset_groups" using the set_reset_group command. However, we use below cmd to create groups automatically. We can also define other characteristics as if clocks are off, when reset is active,
 
create_set_reset_group_scripts -output reset_groups.tcl => rdc reset group definitions. All Primary resets are grouped into 1 group, while each functional reset is grouped into it's own separate group.
source reset_groups.tcl => source above file

An ex of env file:

set_hierarchy_separator . => here . is set as hier separator (Default is /)

create_waveform -period 10 -transition {0 5} CLKA_W
create_waveform -period 10 -transition {0 5} CLKB_W
create_clock -waveform CLKA_W clkA => creates clk on clkA pin/port/net with waveform CLKA_W
create_clock -waveform CLKB_W clkB

create_reset -interval 10 -low -async {reset_n} => specifies "reset_n" is primary reset (default), and is active low. -interval specifies the length the reset to be held active and should be set to lowest possible length (i.e. number of clock periods) that meets the design requirements. Here, it's set to 10 clk cycles. We can also specify waveform of reset pin using -waveform, but most of the times it's not needed, as it's simple state change. -async specifies that the waveform of this reset is to be automaticaly generated by the tool and is async to all other waveforms (generally true). -waveform and -async are mutually exclusive, so only 1 of them should be used.

create_reset -functional -low [get_object_name mod1/sync2/Q] => specifies sync2/Q is an internal reset, which is active low (by default it's active high)

create_input -async {PI_7} => associates waveforms for primary i/p. However, when we don't want to specify a waveform using -waveform, we use -async to indicate that waveform on this object is to be automatically created by the tool, and should be async to receiver (i.e capture clock waveform).

create_input -waveform CLKA_W {PI_1} =>Here we specify a waveform on PI_1.


set_constant -value 1’b0 jtag_pin_1 => All remaining i/p ports are assigned constant values (instead of waveforms)
set_constant -value 1’b1 mod1.gmi_mode => constant can also be set on internal nets and pins.

set_initial_value -interval 10 -value 1'b0 PI_3 => sets initial value on an i/p or inout port for specified interval. At end of interval, signal can take any value at any interval of time (independent of clk edge). This cmd is useful for design inittialization, but is seldom used, as usually we have waveform specified on i/p pin (clk and reset pins are enough)

set_stable_value PI_EN => makes a given port or net's value unchanged from first value that it gets. Thus it's different than above cmd, as we do not specify it's value, but is derived based on deign. After that, the value doesn't change. Thus this cmd excludes given signal from verification, as the signal remains stable, so is not verified during reset analysis. Used frequently for dft signals to cut down on unnecessary noise in results

 set_reset_scenario -name rst_scenario_1  { {reset_n {reset {@t0 0 ->t1} }} {reset_core {reset {@t1 1} }} } => this cmd is used to define resets that work together with other constraints. One command per scenario is needed. While processing a reset scenario, the resets are asserted and de-asserted as specified in the scenario. All reset signals that are not specified in the reset scenario being processed are considered to be in an unknown state (i.e. X). Here, we give the scenario a name "rst_scenario_1". Then we specify the behaviour of reset signals in this scenario. We specify behaviour of each reset signal. Any reset signal not defined in this scenario has a value of X. Here 2 reset signals: reset_n and reset_core are specified with thier behaviour. PI reset_n is specified as reset signal (signal can be either reset or constraint type), @t0 indicates an event at time 0, when the signal "reset_n" changes from x to 0. Arrow -> defines this event (when reset_n changes from x to 0) as t1, so that it can be used in other reset scenarios. Similarly PI reset_core is specified as changing from x to 1 at time t1 (which happens to be time 0). Other special characters as ^, v, # and $ are also used to specify delays. First edge of reset signal is treated as an assertion of reset for metastability checks. Subsequent edge of reset signal is treated as deassertion, but it doesn't matter for metastability checks as deassertion do not start metastability. We can have as many reset scenarios as we want by specifying "set_reset_scenario" multiple times.

set_rdc_false_path -from_scenario rst_scenario_1 => false path from specified scenario to all other. So, no RDC violations from scenario "rst_scenario_1" will be reported.

3. analyze intent => This step checks for correctness of spec provided in setup files. If step was not run, then an env file is created automatically via this cmd.

analyze_intent => all spec such as clocks, resets etc checked. Any problems reported under "MRDC_SETUP_CHECKS" Rule group

4. verify rdc => This is the step that does RDC verification of design.
verify_rdc => rdc structural analysis. Any problems reported under "MRDC_ANALYSIS_CHECKS" Rule group

#verify_rdc -db_model dut_block1-1-2-3-4 => When run with option -db_model, mrdc creates a reduced db with name=dut_block1-1-2-3-4, that doesn't have all design, but only that part that is needed to efficiently run rdc on this block. This is needed when we are running hierarchical flow (discussed below), which uses this lower level reduced db.

#read_rdc_db -block dut_block1-1-2-3-4 -name dut_block1-1-2-3-4 => This reads above created reduced db with name=dut_block1-1-2-3-4.db, for block=dut_block1-1-2-3-4 (dir=meridian_project/block_name/dbs/db_name.db).

source waivers_file => sources any waivers provided

report_policy -compat ALL -verbose -output full.rpt => reports for setup checks, waivers, analysis checks

5. exit => typing exit, quits the tool

Hierarchical flow for mrdc:

Here instead of running mrdc on top level design which may be very large and take days to run, we run mrdc on lower level blocks, dump the reduced db for these blocks. Then we run mrdc at top level, where we read these reduced db for lower level blocks. Tis saves significant run time and makes analysis easy to debug at top level, since lower level module issues are already cleaned up by this time (when we ran mrdc at block level, we try to get it clean, before running mrdc at top level). There may be some differences in violations being reported at top level hierarchical vs top level flat, as some design info is missing when we abstract models of lower level modules. All env files specified when running block level rdc, should be provided exactly the same when running top level runs, so that the constraints are the same, else the model may not reflect the right behaviour (as the model was created in the first place with those set of constraints, so if the constraints changed, then that model is not valid anymore).

IP abstraction: The same concept of hier run is applicable to IP for which we have verified rdc at IP level, and then we incorporate these IP in our design. We can create reduced db for these IP, and then use these reduced db when running rdc at higher level. We blackbox these IP or lower level modules when reading top level design, and then read their respective reduced db files only.

Top level hier run flow steps: Read design files for top level, then read rdc for block level, then provide constraints, and then analyze top level design

analyze -f chip_top.f => reads top level design

elaborate -black_box {dut_block1} => during elaboration, we blackbox lower level modules, since we are going to read their reduced db, instead of reading the whole design for these lower level modules

read_rdc_db -block dut_block1-1-2-3-4 -name dut_block1-1-2-3-4 => reads reduced db for dut_block1 with parameters 1,2,3,4

read_rdc_db -block dut_block1-5-6-7-8 -name dut_block1-5-6-7-8 => reads reduced db for dut_block1 with parameters 5,6,7,8

read_sdc, read_env, analyze_intent .... => Now run the flow as regular flow, and analyze results

-----------------

Details

Published: Wednesday, 15 May 2019 18:48

Hits: 843

Before we get into complicated stuff, let's start from the simplest buliding bllock, atom. Section 3 of Feymann lectures deals with quantum physics that we deal with when talking about atoms. Feynman has one of the best material on Physics.

http://www.feynmanlectures.caltech.edu/

ATOMS:

Atom is the smallest entity that exists for any material. It has neutrons and protons in the nuclues, and an electron circulating around it in orbit. These would seem like particle, but modern quantum theory states that quantum scale objects have wave-particle duality, and exhibit both wave like and particle like properties. There have been many controversies over this wave-particle nature of objects, but is now generally accepted. Bigger the size of object, it's wave characteristics become so small, that it acts like a particle fr most practical purpose. However, for very small objects, as electrons, wave characteristics are noticeable. Similarly all electromagentic radiation (i.e light, heat), etc are assumed to hold both wave and particle characteristics to help explain many phenomenon which can't be explained by assuming only a particle mode, or only a wave model.

For our purpose, we will treat all electron, proton and neutron as particles, with the understanding that they show wave properties. Electrons have -ve charge, protons have +ve charge, and neutrons are chargeless. All of them have mass, but their mass is too small (neutron/proton have 1800 times greater mass than electron, but still small). So, gravitational forces between these masses is too small. They are held together via electrostatic forces (forces b/w opposite charges).

electron: mass = 9.1 x 10^-31 kg = 0.0005amu, charge = -1.6 x 10^-19 C, (electron lot smaller than neutron/proton, so mostly viewed as wave)

proton: mass  = 1.7 x 10^-27 kg = 1.007amu, charge = +1.6 x 10^-19 C, 

neutron: mass = 1.7 x 10^-27 kg = 1.008amu, charge = 0 C,  (neuton slightly heavier than proton)

In any atom, neutron and proton are in center nucleus, while electron is in circular orbit. Number of protons and electron are equal, resulting in 0 charge for an atom. This is also called "atomic number" of atom (num of electrons).

With any atom, energy is needed to break it apart into electrons and protons, That means some energy was supplied to form an atom (electron and proton joined to form an atom). They were formed right after the "Big Bang" when our known universe originated from pure energy some billions of years ago. The energy was converted to the elementary or sub atomic particles (quarks, gluons, leptons etc...). These condensed to form electrons and protons. Very soon, a proton and an electron met, and formed a hydrogen atom (no neutron, so mass=1amu). The neutron came about in a similar manner, and condensed with neighboring electrons and protons to form a complex mixture of the isotopes of hydrogen and helium (helium has 2 neutrons, 2 protons and 2 electrons, so mass=4amu). From this sprang everything else. That is why we find hydrogen and helium in abundance everywhere in universe, since these were the earliest and rudimentary atoms formed.

We can calculate total energy present in any isolated atom, by calculating the electrostatic energy b/w proton and electron, and kinetic energy of electron. We postulate that this total energy has to be -ve, since we have to supply energy to it, to break it. If total energy was greater than 0, then electron would be free to roam around, w/o being bound to the proton.

Electrostatic potential energy: Two charges at infinite distance are assumed to have 0 potential energy. they have attractive/repulsive forces depending on polarity of charges. Let's assume there are 2 charges +q and -q at infinite distance. They have 0 potential and 0 kinetic energy. They start attracting each other because of attractive force. Since Work=F*distance*cos(theta), they do work W=∫ F*dx = ∫ k*q1*q2/r^2 * dx. Integrating this we get W= - k*q1*q2/distance, which represents the final electrostatic potential energy stored in the system of these 2 charges, as +ve work was done on the +q charge. This work done didn't get lost anywhere, and by conservation of energy has to be stored somewhere. The +q charge will finally have some velocity and acceleration (due to force F acting on it), but the final velocity doesn't account for all of the energy. Since the system had 0 initial energy (at infinite distance and zero velocity), final energy also has to be 0. Since final kinetic energy is +ve, an equivalent -ve energy has to be there, which is what is stored as electrostatic potential energy.

When electron and proton come together from infinity to form an atom, we see that the final kinetic and potential energy is not 0. Again, this is expected, since they formed a bond by giving out energy, and so total energy goes -ve. This means to get this atom to have 0 net energy, we need to apply a +ve energy of that value, and then this atom will break apart into an electron and proton. Let's do this calculation for hydrogen atom.

Bohr's model: Very early, Bohr gave a simplified model of what radii, electron are circulating around nucleus. Ground state is the lowest state available for electron which is at radii R = 5.3 * 10^-11m = 0.53A. This radius can actually be calculated by getting total energy of atom (PE+KE), and then minimize this energy. PE decreases as electron gets closer to proton, but KE increases because of velocity increase (required due to uncertainty priniciple, which states that as electron gets coser, uncertainty in it's position is reduced, which requires corresponding increase in velocity to keep the product above the limit). It will turn out that min energy happens at radius R=0.53A.  Look at Feymann's lectures III-02 on the derivation.

centripetal force for electron to revolve around proton is provided via electrostatic force. F = m*v^2/R = k*q^2/R^2 => m*v^2 = k*e^2/R (e is the charge of an electron)

KE = 1/2*m*v^2 = 1/2*k*e^2/R

PE = -k*q1*q2/R = -k*e^2/R

Total energy = KE+ PE = -1/2*k*e^2/R = -2.2*10^-18J = -13.6eV (NOTE: KE is 1/2 of PE but of opposite sign. If KE was same as PE then net energy would have been 0)

1eV=1.6*10^-19J since Force F = q*E = q*V/d (For electric field E= V/d, where V=potential difference b/w 2 points and d=distance b/w them). Then Work W=F.d=q*V/d*d = q*V = 1.6*10^-19*1V = 1.6*10^-19J = 1eV

So, it will take 13.6eV of energy to ionize an hydrogen atom (i.e remove an electron from ground state).

-13.6eV is the lowest possible energy state. An electron can't get any closer than this to the nucleus. An electron can get into larger radius, but then it's speed will decrease (in accordance with hisenberg uncertainty principle). The total energy in this case will increase (though will still be -ve). It turns out that electron can only be in certain discrete radius (i.e it's not possible for it to be in any arbitrary radius). These distinct radius gives rise to distinct energies. We call these different enery levels. This is what is referred to as quantum physics, as electron energy levels are quantized. Of course the highest energy can be anything infinite, but at that point electron and proton are separate and electron is moving at infinite speeds. For our purpose, highest energy level is 0, which is the state where electron and proton just separated, and electron is at rest (no KE). We call this energy level Einfinite. The lowest energy level (-13.6eV) is called E0 (ground state), next one is called E1 and so on (highere states called as excited states, which they get to by absorbing energy from outside. 2 primary ways to excite an electron are thru absorbing light and thru collisions) . In energy state E100 or higher, electron is so loosely bound to the atom, that any small perturbation or external energy will pull the electron out.

En=-E0/n^2 where n is the energy state = 1,2,3 ...

So, E1=-E0/4 = -3.4eV (i.e if electron was in this state, ionization energ = 3.4eV only)

Periodic table shows atoms with 1 electron, all the way to atoms with 100 or more electrons. Each of these atoms consitututes a different element, and that is all the elements we know about. Interesting that just having a different number of electron changes the material so drastically (i.e aluminum vs silicon). Electrons can be in state 1s, 2s, 2p, 3s, 3p, 4s, etc. s states can have max of 2 electrons, while p states can have max of 6 electrons. 1s states correspond to n=0, 2s/2p states to n=1, 3s/3p to n=2 and so on. For a simple atom like hydrogen which has only 1 electron, all states in same "n" have same energy. i.e all states in 3 (3s,3p,3d) have same energy E2=-13.6/9=-1.5eV, However for atoms that have multiple lectrons, these electrons interact with each other, giving rise to splitting of energy levels within same n. So, 2p states are little higher on energy diagram than 2s states, but still lower than 3s states. Detailed diagrams in II-19 of Feymann's lectures. This is why s states are occupied before p states, as electrons always occupy lowest energy states. One important observation is that from states 3s and beyond, energy states start getting closer to each other (E=E0/n^2), which causes many of these energy states to overlap each other. This results in 3d being at higher energy than 4s states. This is why 4s states are occupied before 3d states, as electrons always occupy lowest energy states. More complicated maths involving schrodinger eqn is needed here.

For helium, which has 2 electrons, E0=-24.6eV (so 24.6eV needed to ionize it, or remove 1 electron). If we want to ionize it further, by removing the last electron, ionization energy=-54.4eV. So any further ionization always requires more energy.

For silicon, which has 14 electrons, ionization energy = 8.1eV. Ionization energy is b/w 5-10eV for most elements except 10 elements in 1st and 2nd row of periodic table (since electrons are further out, so lower energy needed).

Collection of atoms:

Above theory applies to isolated atoms. However, when 2 atoms come close together, there is an interaction between the orbits of their electrons. This interaction causes a splitting of each individual energy level into two slightly different levels. The atoms in almost every crystalline solid are so close together that the energy levels produced after splitting due to interaction between the various orbits of different electrons will be very large and so close together as to form a band.

Consider imaginary formation of a diamond crystal from isolated carbon atoms. Link: http://www.engineeringenotes.com/electrical-engineering/conductivity/energy-bands-in-solids-conductivity-electrical-engineering/33944

Each isolated carbon atom has an electronic structure 1s2 2s2 2p2 in the ground state. Each atom has available two 1s states, two 2s states, six 2p states and higher states. If we consider, N atoms, there will be 2N, 2N and 6N states of type 1s, 2s and 2p respectively. As the interatomic spacing decreases, their energy level split into bands, beginning with the outer shell, i.e., n = 2. As the 2s and 2p bands grow, they merge into a single band composed of a mixture of energy levels. This band of 2s-2p levels contains 8N available states.

As the distance between atoms approaches the equilibrium interatomic spacing of the diamond, this band splits into two bands separated by an energy gap Eg. The upper band (conduction band) contains 4N states, as does the lower band (valence band). Thus, apart from the low lying and tightly bound 1s levels, the diamond crystal has two bands of available energy levels separated by an energy gap Eg wide which contains no allowed energy levels for electrons to occupy. These energy bands are actually made up of many discrete energy levels which are too close together to resolve. Within a band the number of levels is of the order of the number of atoms in the crystal, so although electrons are actually restricted to these energies, they appear to be able to take on a continuum of values. 

The lower 1s band is filled with the 2N electrons which originally resided in the collective 1s states of the isolated atoms. However, there were 4N electrons in the original isolated (n = 2) shell (2N in 2s and 2N in 2p states). These 4N electrons must occupy states in the valence band or the conduction band in the crystal. At 0 K the electrons will occupy the lowest energy states available to them. In the case of diamond crystal, there are exactly 4N states in the valence band available to the 4N electrons. Thus at 0 K, every state in the valence band will be filled, while the conduction band will be completely empty of electrons.

In an insulator and pure semiconductor, lower band is completely filled and the upper band is completely empty. The energy of the forbidden gap is denoted by Eg. The conduction takes place only when the electron in valence band jumps to the conduction band. In other words, the electron in valence band requires energy equal to Eg to jump to the conduction band. When the electron jumps from the valence band to the conduction band, then a vacancy electron called a hole is created in the valence band. Since hole is a deficiency of an electron and hence is positively charged. The forbidden energy gap in an insulator is of the order of 5 to 10 eV. The amount of energy cannot be imparted to the electrons in the valence band and hence the electron cannot jump from the valence to conduction band. Therefore, conduction is not possible in the insulators. The forbidden energy gap in case of semiconductor is usually, of the order of 0.75 to 1 eV. This amount of energy can be easily imparted to the electrons in the valence band by thermal agitation of the crystal lattice. Thus, with the increase in temperature, many electrons from the valence band acquire the required amount of energy to jump to the conduction band and these results in the increase of electron hole pairs. The forbidden energy gap Eg is the energy required to break the covalent bands so as to make the electron free for conduction. In a conductor, this bandgap is of order of 0.01eV, so virtually all electrons in valence band avilable for conduction.

 Very good explanation on spliting of energy bands => https://www.youtube.com/watch?v=-lHXZk5M6cI