Directory.Delete() not deleting the empty directory

In C# am having problem in deleting the directory. The directory contains sub directories that have some files.
Now on button click I am first deleting the files, then the sub directory and then the parent directory.
I am getting an error while deleting the parent directory as "directory not empty", however the files and sub directory are successfully deleted.

How to get rid of this ?